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Foreword 
by Vint Cerf 


For nearly a decade, the Future of Text group has focused on interactions with text as largely 
a two dimensional construct. The interactions allowed for varied 2D presentations and 
manipulations: text as a graph, text with appendices for citation and for glossaries, text 
filtered in various ways. In the past year, the exploration of computational text has taken on a 
literal new dimension: 3D presentation and manipulation. One can imagine text as books to 
be manipulated as 3D objects. One can also imagine text presented as connected components 
in a 3D space, allowing for richer organization of context for purposes of authoring, 
annotation or reading. The additional dimension opens up a richer environment in which to 
store, explore, consume and create text and other artifacts including 3D illustrations and 
simulated objects. One can literally imagine computable containers as a part of the “text” 
universe. Active objects that can auto-update and signal their status in a 3D environment. 
Some of these ideas are not new. The Defense Advanced Research Projects Agency (DARPA) 
funded a project called a Spatial Database Management System at the MIT Media Lab in 
which content was found in simulated filing cabinets arranged in a 3D space. One “flew” 
through the information space to explore its contents. What is new is the development of high 
resolution 3D headsets that have sufficiently high resolution and sensing capability so as to 
eliminate earlier proprioceptive confusion that led to dizziness and even nausea with 


extended use. 


The virtual environment these devices create permit convenient manipulation of 
artefacts as if they existed in real space. One of the most powerful organizing principles 
humans exhibit is spatial member. We know where papers are that are piled up on our desks 
(“about three inches from the top...”). VR environments not only exercise this facility but 
also allow compelling renderings of information, for example, highlighting relevant text 
objects in response to a search. Imagine walking in the “stacks” in a virtual library and 
having books light up because they have relevant information responsive to your search. One 
could assemble a virtual library of books (and other text artefacts) from online resources for 
purposes of preparing to engage in a research project. Could we call this an information 
workbench or machine shop? Because of the endless possibilities for rendering in virtual 
three-space, there seem to be few limits to a textual “holodeck” in which multiple parties 


might collaborate. 
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We are at a cusp enabled by new technology and techniques. The information landscape is 


open for exploration. 


Vint Cerf @ The 11th Future of Text Symposium. Hegland, 2022. 
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Welcome 
by Frode Hegland 


Along with Vint Cerf, Ismail Serageldin, Dene Grigar, Claus Atzenbeck and Mark Anderson I 
welcome you to ‘The Future of Text’ Volume 3, where we focus primarily on text in virtual 
environments (VR/AR) and text augmented by AI. In other words, text in 3D and text in 
latent space. This volume of The Future of Text includes: 


¢ Presentations from the 11th annual Future of Text Symposium held on the 27th and 
28th of September 2022 online and at The Linnean Society in London, either as 
transcripts, articles independent from presentations or presentation notes. Where 
presenters used images, they have largely been included here. No copyright infringement 
intended. Where there is an issue of rights, please contact us. 
https://thefutureoftext.org 
¢ Articles from our Journal & Transcripts from Monthly Presentations. 


https://futuretextpublishing.com 


The hope is that this work will inspire you to think richly and deeply about a future 
where text is freed from the traditional flat rectangle. Soon we will live in a world where 
VR is just part of our daily experience. We have a brief opportunity left to dream of what this 
can be before big companies release their headsets and realise some of this potential. We now 
have an obligation to use the power of our imagination to think of alternative futures, un- 
clouded by the corporate implementations. Together, I think we can dream of amazing futures 


which can inspire future generations who will have lived with VR all their lives. We start 


with a slightly paraphrased quote from a relatively obscure Apple Macintosh commercial! 
from the 1990s: “The only limits will be the size of our imagination and the degree of our 
dedication.” Thank you for being a part of this journey. We can only improve the future of 
text if we do it together. 


Frode Alexander Hegland | frode@hegland.com | Wimbledon, UK 2022 
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Our work in VR 


We built a few VR experiences and experiments for the Symposium, listed below, which 
should work on most headsets since they are web based. Please feel free to give any feedback 
by clicking on the person’s Twitter handle. Dialog is so important but please remember these 
are experiments and not polished final experiences. 

‘Simple’ Mural A simple and powerful introduction to VR, this shows a single Mural 
by Bob Horn, which you can use your hands to interact with: Pinch to ‘hold’ the mural and 
move it around as you see fit. If someone says VR is just the same as a big monitor, show 
them this! 
https://zachernuk.neocities.org/2022/nirex-mural/ (By Brandel: https://twitter.com/zachernuk) 

Basic Author Map of the Future of Text Open this URL in your headset and in a 
browser and drag in an Author document to see the Map of all of the contributors to The 
Future of Text book. 
https://t.co/nEIoUpiUsW (By Brandel: https://twitter.com/zachernuk) 

Self Editing Tool In this environment you will be able to directly manipulate text and 
even execute the text as code by pinching these short snippets. Fabien recorded a 
walkthrough video here: https://video.benetou.fr/w/ok9a1v33u2vbvczHPp4DaE 


https://t.co/oTCII9zxLx (By Fabien: https://twitter.com/utopiah) 


Simple Linnean Library A rough and ready room made by a novice, this is something 
you can also do. I used Mozilla Spoke to build an experience which can be viewed on any 
browser, in 2D or VR in Mozilla Hubs: https://hubs.mozilla.com/spoke 


https://hubs.mozilla.com/Wun7r4m/distinct-mild-plane (By Frode: https://twitter.com/ 
liquidizer) 
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This Book as Augmented PDF 


This book is available in printed form and as a PDF document with ‘Visual-Meta’ metadata, 
developed by the people behind The Future of Text. If you choose to read it in our free 
‘Reader’ PDF viewer for macOS (download’), you can interact with the document in richer 


ways than you normally could. 


* the ability to fold the journal text into an outline of headings (cmd-) 

* pop-up previews for citations/endnotes showing their content in situ (simply click on 
them) 

¢ doing Find, with text selected, locates all the occurrences of that text and collapses the 
document view to show only the matches, with each displayed in context (cmd-F) 

¢ if the selected text has a Glossary entry, that entry will appear at the top of the screen 
when doing a Find command 

* copy text which will be pasted as a citation if using a Visual-Meta aware word processor, 


such as out own ‘Author’ :https://www.augmentedtext.info 


You can read more about what Visual-Meta brings to metadata here: visual-meta.info This 
work will also be made available in other formats, including .liquid and JSON for the 
purposes of developing text interactions, please get in touch if you would like any of these 


formats. Reader can be downloaded for free here: 


https://apps.apple.com/gb/app/reader/id1179373118 
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Editor's Introduction 


VR (including AR) is about to go mainstream and this has the potential to offer 
tremendous improvements to how we think, work and communicate. 

There are serious issues around how open VR work environments will be and how 
portable knowledge objects and environments will be. Think Mac VS. PC and the Web 


Browser Wars but for the entire work environment. 


The potential of text augmented with AI is also only now beginning to be 
understood to improve the lives of individual users, though it has been used in various 
guises and under different names (ML, algorithms, etc.) to power fantastic services (speech 
understanding, speech synthesis, language translation and more), as well as social networks 
and ‘fake news’ for years. 

More important than the specific benefits working in VR will have, is perhaps the 
opportunity we now have to reset our thinking and return to first principles to better 
understand how we can think and communicate with digital text. Douglas Engelbart, Ted 
Nelson and other pioneers led a ‘Cambrian Explosion’ of innovation for how we can interact 
with digital text in the 60s and 70s by giving us digital editing, hypertext-links and so on. But 
once we, the public, felt we knew what digital text was (text which can be edited, shared and 
linked), innovation slowed to a crawl. The hypertext community, as represented by ACM 
Hypertext, has demonstrated powerful ways we can interact with text, far beyond what is in 
general use. Still, the inertia of what exists and the lack of curiosity among users have made it 
prohibitively expensive to develop and put into use new systems. 

With the advent of VR, where text will be freed from the small rectangles of traditional 
environments, we can again wonder about the possibilities. This will unleash public 
curiosity as to what text can be once again. 


To truly unleash text in VR we will need to re-examine what text is, what 
infrastructures support textual dialogue and what we want text to do for us. The 
excitement of VR fuels our imagination again — just think of working in a library, where 
every wall can instantly display different aspects of what you are reading, having the outlines, 
glossary definitions and images from the book framed on the wall, all the while being 
interactive for you to change the variables in diagrams and see connections with cited 


sources. This could be inspiring or distracting but the key is you can change it at a whim. 


This is an incredibly exciting future once headsets get better (lighter, more 
comfortable, as well as better visual quality). Because this cannot happen without 


fundamental infrastructure improvements, what we build for virtual environments—V R—will 
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benefit text in all digital forms. This is important. 


The future of humanity will depend on how we can improve how we think and 
communicate and the written word, with all its unique characteristics of being 
swimmable, readable at your own pace and so on, will remain a key to this. The future of 


text we choose will choose how our future will be written. 


Why VR, Why Now? 
My starting position is that VR, sometimes also called ‘metaverse’ this days and ‘cyberspace’ 
before, is about to go mainstream. 

This is based on Meta Quest 2, which is available for the mass market and currently 
out-selling the Microsoft Xbox game consoles. It is just the start of what VR headsets will be 
able to offer. The view inside such a headset is already rock-solid, whatever environment is 
present, it looks like it is there, right in front of you. With Apple’s headset coming next year 
and improvements coming along as we have seen with personal computers, smartphones and 
smartwatches, this will rapidly continue to improve to the point where the visual fidelity 
becomes high and the discomfort low. 

The future is coming fast. It is worth emphasising that in the same way the room sized 
computer was not really a clear precursor to the smartphone, the current bulky, low resolution 
and narrow field of view devices does not illustrate what in the near future will feel 
lightweight and the visual quality will approach photo realism—it will feel like the world is 


transformed-—it will not feel like we are wearing a heavy headset. 


What this will unleash we do not know, but what I do know is that we, as a wider 
community of authors and readers of text, need to get involved in thinking about—dreaming 
and fantasising—about what it can be. For starters, we will not be using headsets all the time, 
any more than we now only ever use a smartphone or a desktop/laptop. We will enter VR 
when we need to focus on something, similar to how we enter a movie theatre, or turn on a 
large, flat screen TV when we want to be immersed or watch general video ‘content’ on all 
our devices. 

The distinction between VR and AR will likely become different modes on the same 
device but will have very different uses. Where AR refers to the world, VR will refer to any 
world. There is also an interesting middle ground, where the view of the world is superfluous, 
and it is just there for a sense of place, where the knowledge objects being interacted with are 
in a space, and the background could be anywhere. This is demonstrated in Yiliu Shen- 
Burke’s work where the user can interact with a constellation of knowledge, and the 


background is simply a background, even though it is a live video of the user's room. There is 
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also what is referred to as ‘reverse AR’ where the whole room environment is synthetic but 


the main object in the room is real, as built by the team at Shopify to let shoppers try a chair 


and then look at the room as though they are at home’. There is a lot of creativity as to where 


boundaries will be and it will only become more and more interesting. 


We had a historic opportunity to re-think text in the 1960s, and now we have another. 
This is a once in a lifetime, once in a species point in time. We are only a few years away-if 
that-from VR headsets becoming commonplace. The dreams of Doug Engelbart and Ted 
Nelson, among other true pioneers, have not had a place to put their feet over the last few 
decades. There has not been a foundation of need for improved text interaction from people. 
Now there is. With VR, it’s easier to see that there are new ways of working. Quite simply, 
we have an opportunity to dream again. ‘VR’ won’t be ‘VR’ for long, same as ‘hypertext’ 


became the web then became just ‘online. ‘VR’ will become ordinary very soon. 


Why AI, Why Now? 

The further assumption is that AI will continue to advance. We are looking at is the 
emergence and improvement in automatic pattern recognition, classification, summarization, 
extrapolation, and natural language query-based information extraction for everything from 
speech to text and text analysis. We are also keeping an eye on the development of Self- 
Aware Artificial General Intelligence with a mixed-initiative conversational UI, since it never 


hurts to dream far into the future. 


Al, if left unchecked, can present real dangers for society, as seen already in the basic 


AI algorithms which shape social media interactions and more. 


AI can expand our understanding of creative expression. In this volume we have the 


experience of Mez Breeze who explores the art of AI and associated text-driven potentialst. 


One useful way to think of AI is as a digital map. I came to think of this when my 5 
year old son started navigating for us when driving in Norway this summer. Since the map 
was not un-augmented paper but a digital map on an iPhone, he was helped by always 
knowing our location and there was always a blue line suggesting where we should go, so he 
could tell me ‘right’, ‘left’ and what exit to take off a roundabout, in his youthful happy 
voice. The map did not dictate where we went, we could always choose a more scenic route if 
we felt like it, and the blue line would update its suggestions. 

More than anything, AI has been largely ignored when it comes to text. The Apple 
Watch I use I can rely on to accurately understand my commands, which is quite mind- 
blowing. I have refined speech to text in my macOS word processor ‘Author’ to take 


advantage of Apple’s increasingly powerful API. Some software provides coloured grammar 
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when required and some suggest changes to writing style. There are of course relatively brute 
force AI analysis of masses of academic documents and there are writing tools which will 
write based on supplied text, such as GPT-3, but I suspect, this is really just the snowflake on 


the top of the iceberg of what is possible. 


What live analysis can a knowledge worker hope for when writing? How about hitting 
cmd-? and getting a list of suggested next paragraphs (not the less-then-helpful-help-menu). 
Maybe there are a few suggestions, one based on what the author has typed so far and the 
author’s own body of work, one based on what’s typed so far but including all known 
documents in the author’s field and a third maybe also including what’s found on the web? 
This is the digital map approach, giving the user guidance, but not dictating. This is work 


currently undertaken by Pol Baladas on Fermat, for example. 


Al is both ‘just beyond the horizon’ and also becoming mundane so it is valuable to try 
to understand, then to revise our understanding, of how AI can augment our interactions with 


text. 


The Future of Us, The Future of Text 


2022 is the year of a continuing pandemic, along with economic collapse, inequality, a 
significant war in Europe which threatens the stability of countries near and distant, as well 
as the underlying climate change catastrophe we are now seeing starting to make an impact 
on our daily lives. 

There is no question that if we are to survive, let alone thrive as a species, we need to 
improve the way we communicate and relate to each other. This will mean looking at how we 
can improve education, politics, scientific discourse and even how we can bring our spiritual 
practices into play to improve, quite simply, how we get along as people, how we develop 
shared goals and how we deal with conflict. 

Much of dialogue, from politics, law and international treaties, to social media, lab 
reports, journal articles and personal chat, is in the form of text. I believe that we have to 
improve how we interact with textual knowledge, otherwise, we will be manipulated by those 
who do, such as social media companies and we will continue to be overwhelmed by the 
sheer volume of information. We cannot rely on face-to-face speech and video alone. We 
have to improve what text is, how we can interact with text and how we can represent text. 

From its invention almost five and a half thousand years ago, the written word has 
proven remarkably powerful in augmenting how we think and communicate. The transition to 
digital text has transformed text, a medium which before becoming digital was primarily 


about fixity, about thoughts being securely placed on a substrate. When text became digital, 
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this attribute largely vanished, with text now being interactive. A user could easily delete any 


text, cut & copy and edit the text freely, giving text a much more fluid character. 


What was initially a revolution with the editabillity, and soon after the linkability of text 
became part of our daily lives, the magic of what was previously referred to as ‘hypertext’ 
simply became ‘text’ and analog text, previously only referred to as ‘text’ became ‘print out’ 
or ‘hard copy.’ The magic of digital text became mundane. 

Other digital media continued to develop however. This was all the while digital images 
went from wireframes to photorealistic and games went from abstract ‘asteroids’ to deeply 
immersive and interactive experiences. We collectively thought we knew what text was, and 
little innovation took place. However, as digital text proliferated at an astounding pace, 
overwhelming those trying to stay on top of research, social media companies and those 
seeking to influence popular and political opinion went to work creating powerful tools for 
textual persuasion. We got social media echo chambers with algorithms designed to provoke, 
to increase ‘engagement’ (and thus ad views resulting in greater revenue) and modern ‘fake 
news’ at the start of the war in Ukraine in 2014, when Russian intelligence flooded digital 
mass media and social networks with fake and real news to the point where it became 
difficult to discern what was actually going on. Fake news continued to influence people’s 
opinions at the same time as research documentation stayed hardly digital, with little 
interactions afforded to the user. There are many issues to be discussed in the paragraph and 
I'd be very happy to go through them in person, but the point is simple: Text interactions 
became sophisticated where there was an incentive to invest in it in the form of money and 
political control. Where the greatest benefit to the end user could have been seen, there has 


been little innovation or investment. 


We had a historic opportunity to re-think text in digital form but we dropped the ball. 
We don’t have the ability to ‘fly through cyberspace.’ We have the ability to cut and paste in 
Word, click on one-way, one-destination, un-typed links and edit a document together in 
Google Docs. We could do more, much more. We could imbue all documents with rich and 
robust metadata. This is a personal issue for me. We could provide authoring and reading 
software as powerful as Apple Final Cut. We could have reached for the stars, but the market 
and the few companies making text-focused software decided on ‘ease of use,’ and we were 


left with big buttons to click on. 
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Improving not only VR Text or AI Text, but ALL Text 
It is important to point out that the opportunity is not just about working in VR or using AI 
augmented text. 

The real opportunity is that we will have an opportunity to rethink everything with 
digital text because the public’s imagination will be energised—all text can benefit from a re- 
think and new dreaming. 


It is clear that while text in documents will continue to matter, it will not just be text 
‘floating in space’. 

It is also clear that better metadata will make text more usefully interactive on 
traditional digital displays as well. 

This is a historic opportunity primarily because we can restart and think from first 
principles: how to connect people and how to help us think with symbols/text. Our planet and 
our species is facing serious threats so it is important that we learn from the past and that we 


are not shackled by the past. 


We need to look at how we can usefully extend our cognition to better think with other 
minds, as Annie Murphy Paul discusses in her book The Extended Mind (Murphy Paul, 
2021) and in her talk in this book. Jaron Lanier—the man who embodies VR— and who 


presented the keynote at the Future of Text Symposium puts it ‘The solution is to double 


down on being human’.’, 


The solution is at the same time to extend our mental faculties to really take advantage 
of the flexibility of representation and interaction these future environments will offer us. Just 
as we are today hamstrung by being tied to the models of paper documents, we must expand 
our minds in entirely new ways to get the most benefit out of what can now be created. This 
will mean building systems which connect with our physiology to learn to ‘read’ and ‘write’ 
in entirely new ways. Think how text seems entirely artificial if you take a human’s situation 
100,000 years ago, but it seems natural today. Text is only lines on a substrate. What will be 
the future of text when the entire visual, aural—and soon haptic—field can be used for 


expression and impression? 


What does it mean to be In VR? 


Virtual environments will feel more like rooms or full environments than what we think of as 
textual ‘documents’ today. There will be intricate models of microscopic creatures for us to 
explore, we will be able to walk through cities; ancient, modern and futuristic. We will also 
be able to step into space ships and explore entire planets and more. This will be exciting, and 


valuable and it will take teams of people a serious investment in time, energy and money to 
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build these experiences. A great example is the work of Bob Horn who extends murals into 
multiple dimensions which at first glance is just an image shown large in VR but on further 
interactions becomes so much more than it could have been if it was simply printed onto a 


wall. 


We will also have new ways of telling stories, as Caitlin Fisher who works on the 
opportunities for more immersive storytelling in VR discusses in this book. The 
opportunities are vast for what we can be in virtual environments but for this book and this 
project we are looking at text primarily, which will include many types of packages and 


experiences, one of which will remain a kind of book. 


Documents in VR 


One of the key questions we ask is: What is a document in virtual reality, and more 
specifically, what is an academic document in VR and what does it become with AI 
augmentations? 

We look at academic documents as a special case since academia is a field connected 
by documents and it is also a field where what is in the documents needs to interacted with 


and connected. 


This is distinct from commercial books where the owners of the intellectual property 
have reason to restrain the use of the text and is therefore a different strand of the future of 
text, one with constraints outside of what we are currently looking for. We are, by the nature 
of trying to look into the future and wish what might be to augment how we think and 
communicate, dreamers, and as such our playground is information which is free to a large 


extent. 


There are limits to online-only documents which are worth noting, since it is easy to 
consider virtual environments to be online. The first is addressability and the second is 
reliability. Imagine if you could only get a book at the library by knowing it’s location, as in 
its entry in the Dewey Decimal Classification system—and not by the title of the book or the 
author’s name(s). This is effectively what web locations are; you can locate information 
based on location, not by content or metadata. Academic citations, which simply presents the 
documents metadata, such as title, author(s) names and date of publication do not tell you 
where you can locate the document, but what information you need to locate it in many types 
of places, such as libraries and book shops. The second limitation is reliability based on the 
DNS (web domain system) where the documents cease to be available if there is non- 
payment of the DNS fees or if there is any technical issue with a specific server or set of 
servers. Many people exists in a tine sliver time, a few years before ‘now’ and with a few 


vague prods into the future to have an idea of their career advancement, prospective new 
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home, the lives of their children and so on. Academics have to live in much longer timespans, 
almost no matter what field of study. Their research will include ‘up-to-the-minute’ 
knowledge but also access to what’s behind it. Similarly, academic have a duty to the future 
to make their work available long after they are gone. 

Documents for virtual environments can draw on previous types of documents and 
extend them. There is no reason why they should not have the option to be primarily text but 
still have a spoken presenter available if the reader would like to hear a perspective. There is 
also no reason why they should not be compressible into a portable document form like we 
have today. In this volume of The Future of Text, we can see how Bob Stein looks at the 


book’s essence in digitally empowered form and extends large collections of knowledge. 


Metadata Matters. Make it visual, to the same degree that text is 


The more we look at how to realise the incredible potential of text in VR and text augmented 


by AI, the more it becomes clear that better? metadata which is needed to make it happen. 
It is better metadata which augments AI to be able to make better analysis. 
It is better metadata which makes text in virtual environments flexibly interactive. 


Metadata is the data which makes data useful. A basic example is a document which can, but 
in practice in 2022 hardly ever does, contain embedded, or hidden, metadata to make the 
name of the author(s), the title and publication date known. Visual-Meta, developed as part of 
the Future of Text Initiative (and which is also my PhD thesis result) includes this in the 
appendix in as simple a way as ‘author = {Name of Author}’ ‘title = {Name of 

Document}’ ‘month = {September}’ ‘year = {2022}’. This ‘self-citation’ metadata is what 
makes it possible to automatically cite the document, through a simple copy and paste, and to 
see it in a network of other documents, where the metadata is in the document itself and not a 
separated database. 

Visual-Meta is my approach to robust metadata and I highlight it to highlight the issue 
of metadata, it is quite clear that much work needs to be done beyond what Visual-Meta 
enables. 

All the multimedia objects are included in this so that they are flattened into 3D when 
published as a document and can be re-invigorated with all dimensions when viewed in VR. 
This includes spatial information of how the document should be be shown, by default, in VR 
3D space. It also includes all the chart information and image map data. Including image map 
data in the metadata in this way means that a document can contain a huge mural, shrunk 
down to a double page spread in the document, but then it can be viewed wall size, with all 
data and links intact, at will. 


27 


Since Visual-Meta was developed as my PhD thesis, I find I need to come to its defence 
and specify that adding the Visual-Meta appendix to documents is completely effortless for 
the author when the system supports it. What is put into the Visual-Meta is usually metadata 
which the authoring software is aware of, such as headings, glossaries and glossary terms, 
references, and chart and graph information, but this is currently discarded on export/ 
publishing. Visual-Meta simply keeps it and makes it accessible. 

Reading documents with rich metadata included, and working with the documents to 
produce new knowledge, is more flexible and robust: You can choose what to view and you 


do not need to worry about transcription errors or data loss. 


Scale of Change 

Having considered some of the scenarios and aspects of working in a virtual environment I 
hope you might agree that the difference between a laptop screen and working in VR will be 
as large as looking at the world through a small picture frame and putting the frame down and 
looking at the world fully and richly. Personally, I think that, after a while, it will effectively 
be bigger than going from analog to digital, but only time will tell. It will be something new 
and it will be a fundamental part of our lives. “VR will never be the same as physical 


reality... We'll just live life across multiple realities. Each with their own physics, bodies & 


affordances.” says Andreea Ion Coj ocaru>. 


Concerns 

Some of the wonderful potentials above seem almost pre-ordained. But it is not. The only 
thing pre-ordained is that large companies will invest masses of resources to own this new 
environment to create highly profitable cashflows, as this should be. Issues around the use of 
VR, such as how walking around virtually can produce a feeling of nausea for some, but if 
you instead pull objects, such as a massive wall-sized mural towards you with a gesture (such 
as pinch and pull) you will feel fine, even though visually it is the same impressions to your 
eyes. These usability issues is most certainly important and that is why they are being looked 
at by the companies building the VR environments. What the are not focusing on is 


ownership and transferability: 


Ownership & Transferability 


Considering that what is happening is the creation of a whole new world, it is probably not a 
great idea for a few huge companies to own all of it. We need an ‘Internet’ for VR. We need 


open standards so that our information stays free for use, to use as we see fit, and not trapped 
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in a corporately owned framework, as happened with the Microsoft Office formats, for 


example. 


A simple dream would be to work on something on a traditional device, like a laptop, 
and to be able to don a headset, and take that information out of the screen and into the VR 
environment. But how can the VR environment know what is on your laptop's screen and 


how could any changes be communicated back? 


Questions we need to ask include: What would happen if the document/knowledge 
object you worked with in one VR room, where you gave it fancy interactions and powerful 
views, simply won’t render correctly in another room when you try to share it with 
colleagues. It could also happen that we repeat the mistakes of digital text over the last 
decade and have shiny and involving social media text but little to interact with it to help us 
think, only share. We will need open, accessible and robust infrastructures to allow the VR 


world to flourish. 


What We Are Doing 


To help realise the potential of richly interactive text in a virtual and traditional environment, 
text which is directly manipulable and which can be interacted with through AI systems, we 


are doing the following: 


¢ Hosting the annual Future of Text Symposium for over a decade 
thefutureoftext.org 


¢ Hosting the Future Text Lab with twice-weekly Open Meetings & monthly Guest 


Presentations 
futuretextlab.info 

¢ Publishing The Future of Text series of books of which this is volume 3 
futuretextpublishing.com 


* Working on the metadata standard Visual-Meta (robust metadata storage in VR & 


traditional domains which Vint Cerf calls “exploitable self-contained self-awareness?’’) 
visual-meta.info 


¢ Building the Author and Reader software (as Alan Kay—contributor to volume 1|-—said: 


“the best way to predict the future is to invent it”) 


augmentedtext.info 


We also experiment with VR environments, where what we learn from experience continues 
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to surprise us. On the positive side, it is impressive how stable the environments are, much 
more than we expected—when putting on the headset (we primarily use the Meta Quest 2), 
whatever environment we go into, it feels like we are really there, it does not wobble or feel 
off at all. There are small surprises which we need to take into consideration. For example, 
pulling a large mural towards you gives people significantly less motion sickness than if they 
simulate walking up to it, even though the visual display is practically identical. Furthermore, 
having lines in space to show relationships is quite annoying outside of very specific use- 
cases, as it feels almost like physical strings have been placed in your space. Similarly, text 
floating in space without a background can easily become very hard to read. Furniture is also 
an interesting issue since most people don’t have ‘VR Only’ rooms. Therefore the desk, 
chairs and other furniture must be taken into account when designing virtual rooms where the 


user can stand up and move. 


The Bottom Line 
At the end of the day I am asking you, if you are ‘sold’ on the idea that VR, or the 


‘metaverse’ will become mainstream over the next few years, to consider what this truly 
could be to help us think and communicate, to help us work and learn—as well as how you can 
help inspire others to ask the same questions. Then I ask you to consider how we can keep 
this environment open and not as a series of corporate workrooms isolated from each other 


and the rest of our information. 


The Invitation 
In publishing this I am inviting you to join us in dialog about what text can and should be in 
an environment where text can be pretty much anything our imagination points to and 


implementation allows. 


The Dream 


The imagining and dreaming needed will be huge. It is exceptionally difficult to see and 
dream beyond a linear extrapolation of what we experience. We, therefore, need to support 
those who have the capacity to dream, in the spirit of Doug Engelbart, and foster dialogue for 
a broader community to dream together, and not simply fantasise, at a cartoon level, on a 
magic text which has no bearing on implementation. By this I mean purely shifting the act of 
reading and writing to artificial systems to somehow do the work for us. We need to augment 
ourselves, both through removing unnecessary hurdles and reducing clerical work, such as 


the huge amount of effort placed on the cosmetic aspect of citations and formatting for 
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journal articles. 


The infrastructure to support the dreamt-up futures will need to be radically better than 
what we have now IF we want to have an open future for how we can interact with our 
knowledge and each other through the medium of the written word. The substrate of text used 
to be a plain material, such as paper or parchment, but now it is not the screen but everything 
behind the screen; the storage of the type, the metadata which makes the type useful and the 
means through which this can be shared openly and stored robustly. 


31 


32 


Andreea lon Cojocaru 


Borges and Vygotsky Join Forces for BOVYG, Latest Virtual Reality 
Start-up 
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The Future of Text. Cojocaru, 2022. 
Abstract 


Can virtual reality reinvent text, revamp human communication, and chart a new course for 
us all? If there was ever any hope, it is in BOVYG. Investors are flooding in the seed round 
of this promising venture. The Guardian obtained a transcript of a private work session 
between Borges and Vygotsky. The discussion, centered on the process of concept formation 
and the mechanism through which words reflect reality, implies nothing short of a brand-new 
ontology. 


For readers unfamiliar with the work of these two giants, we recommend at least a cursory 
reading of Borges’s Tl6n, Uqbar, Orbis Tertius and Vygotsky’s Thought and Language before 
reading the transcript. 
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Body 


[This transcript is based on a video recording. The capture is from BOVYG, a VR application 
Borges and Vygotsky are developing. The headset recording is Borges’s. We are not sharing 
this in a video format because the visuals are quite uninteresting. The entire conversation 


happens in what appears to be an empty scene with a white virtual box in the center. ] 


Borges: Vyg, this thing — do you see it? What’s this? 

Vygotsky: It’s a box, B. I just put it there. The word “chair” is written on it because it’s 
supposed to represent the word and the concept. Let’s start with the simple stuff today, for a 
change. 

Borges: Vyg, okay, but why are we starting with the end? This VR stuff is supposed to be a 
brand-new start. That’s the whole point, isn’t it? 

Vygotsky: Of course. So we start with word and concept, then we work our way backwards, 
then, hopefully, forwards, and we see how things play out in here. We keep an eye out for 
different turns in the concept formation process. 

Borges: Vyg, please. Look at this box and at this word on it. We are at THE END of the 
concept formation process. The process that got us into this mess to begin with! The world is 
simply not a grouping of objects in space. It is a heterogeneous series of independent acts. 
The world is successive and temporal. Idealized objects like “chair” should not be relied on. 
There shouldn’t be any fixed concepts to begin with. Instead, everything should be invoked 
and dissolved momentarily, according to necessity. 

Vygotsky: B, sometimes I think that this predilection of yours towards subjective idealism is 
taking worrisome turns. Yes, Tl6n, Uqbar, Orbis Tertius was brilliant, and you got them all 
wrapped around your finger. But this is serious work! We are not here to write another five- 
pager on magical realism. In virtual form, but this IS reality. More than that, this is the 
FUTURE of reality! Humans master themselves FROM THE OUTSIDE! The development 
of thinking is from the social to the individual. People first receive language which leads to 
communicable concepts and world views. Language and world formation rely on stable 
concepts, not fleeting impressions that “dissolve”! 

Borges: Vyg, what language do you see in here? This box with letters on it? What do these 
letters mean in here? Where is the chair? Can we sit on it? It is leather? Do we sit on it by 
moving out butts downwards or perhaps upwards? 

Vygotsky: What’s your point, B? Just get to the point! 

Borges: Vyg, there are no objects or concepts, at least no permanent ones. Not in physical 


reality, and definitely not in here. Is a dog seen from the side and then seen from the back the 
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same dog? Only if you rely on thinking processes that manipulate objects called “dogs”! 

Only if you need to — pointlessly if you ask me — extend existence and identity beyond the 
current moment and into some weird — and dangerous! — permanence. It’s all made up, Vyg, it 
really is...! And, in here, the lie is outright unbearable! 

Vygotsky: What do you mean “in here”? What is so different “in here”? 

Borges: Everything! Let’s take this box. Look at it from the side and look at it from the back. 


Is it the same box? 
Vygotsky: Hmmm... 
Borges: No! Of course not! Every second, this box is exactly 90 boxes! 


Vygotsky: B, don’t go all techie on me. The only thing that matters is that we think this is the 
same box. Permanence and identity are necessary NOT fundamental. 

Borges: What are they necessary for, Vyg? 

Vygotsky: We need them to generalize, of course! We think by using concepts, encapsulated 
into words. Think of words as tools. That is how we can build thoughts on top of thoughts, 


using both our own words and those of other people. 


Borges: Vyg, you are describing the labyrinth of abstractions we need to break out of! We are 


here to design the process that breaks us OUT of it! 
Vygotsky: The labyrinth IS the process, B... Perhaps we can shift towards new ways of 


building the labyrinth, but we cannot exit it. There is nothing beyond it... Our functioning as 


human beings relies on this clear framework. You can call it a labyrinth if you wish. 
Borges: This framework of yours, Vyg, is clear. Terribly clear. That’s precisely the problem. 
You forget that we are both Theseus AND the minotaur. As thought become verbal and 
speech becomes intellectual, as you so often like to say, we both trap and chase ourselves 


inside it. [Sighs for a while.] Let’s run this scenario with this box of yours in here. 
Vygotsky: Which box? 

Borges: This one, over here, with “chair” on it. 

Vygotsky: From which side? 

Borges: From this side! 

Vygotsky: Now? 

Borges: No, when I said it a second ago! Or... yes... now as well! 

Vygotsky: From which side? 

Borges: This! 

Vygotsky: Now? 


Borges: Now?! 
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[We pause our transcript here. This almost monosyllabic conversation about the virtual box 
continues for another hour. Then they break for lunch. When they return, the conversation 
continues to be monosyllabic although a clear change in tone indicates that they are now past 
the disagreement related to the box. Our best explanation for this change in communication is 
that, similar to a process often described by Tolstoy, the closeness between the two, in 
combination with the strange affordances of the virtual medium, has enabled them to 


abbreviate their communication to the point where it is incomprehensible to the rest of us. | 


Author's Notes 


Lev Vygotsky (1896 — 1934) was a Russian cognitive scientist, psychologist, constructivist 
and critical realist whose work focused on the internal mental structure of an individual. 
Methodologically, he focused on relationships, processes and levels of analysis. He is best 
known for sociocultural theory, a developmental school of thought focused on the 
relationship between thought and language as independent and dynamic processes in 
ontogenesis, phylogenesis, and within a cultural context. This dialogue speculates on 
Vygotsky’s position regarding language and virtual reality based on his book Thought and 


Language. 


Jorge Luis Borges (1899 — 1986) was an Argentinian writer, essayist and translator known 
for his trademark themes: dreams, labyrinths, libraries, language and mythology. His stories, 
non-linear narratives that mix fact, fantasy, hox and forgery, are generally considered to have 
reinvented modern literature. This dialogue speculates on Borges’s position regarding 
language and virtual reality based on his short stories Tl6n, Uqbar, Orbis Tertius and Funes, 
the Memorious. Moreover, the entire conversation makes use of many of Borges’s literary 
techniques. Most of the time I stay close to what the main characters could have plausibly 
said in such a situation, but, like Borges in his own stories, I also diverge from that and use 
the two characters to purse my own arguments. Hinted at by the fact that the footage was 


recorded in Borges’s headset, this is the kind of thing he would write. 
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Journal Guest Presentation ‘An Architect Reads Cognitive Neuroscience and 


Decides to Start an Immersive Tech Company’ : 13 May 2022 


https://youtu.be/4 YO-iCUHdog? t=678 


Andreea Ion Cojocaru: Hi everyone. It's such an honour to be part of the group, and present 
to this group. Because this group is very different than the usual audiences that I speak to, I 
took the presentation in a very new direction. It's a bit of a risk in that I’m going much deeper 
than I’ve ever gone before in public in showing people the insides of how my method works. 
So part of what you will hear will be the messiness of what is a very active and sometimes 
stressful process for us at Numena. But hopefully, yes, there will be time at the end for you to 
ask questions, and for me to have the chance to clarify the aspects that were maybe a bit too 
unclear. Okay, with that mentioned I’m going to share my screen. All right. I just gave a title 
to this talk. This talk did not have a title until five minutes ago, and now it's called An 
Architect Reads Cognitive Neuroscience and Decides to Start an Immersive Tech 
Company. And this is pretty much what the story will be today. 

I’m an architect. I have a master's degree in architecture. I’ve been in love with 
architecture and the idea of space-making for as long as I can remember. But there's a bit of a 
twist in my background in that, when I was young, I was learning letters by typing with my 
dad on a keyboard in the 80s, and I have this childhood relationship with computers and 
coding. And I’ve always been very passionate about philosophy. So a while back I 
discovered cognitive neuroscience and I began reading that from the perspective of an 
architect who can code and who is also an amateur philosopher. Reading this from this 
perspective and I don't know how many people read content neuroscience with this kind of 
background gave me all sorts of ideas. 

When I discovered AR and VR, and specifically VR, I just found this opportunity to 
start pursuing some of the ideas that have been floating around my mind, in reading cognitive 
neuroscience for a while, this started. So the company started about four years ago, and it's 
been a crazy ride. 


But I’m not going to start with what the company is doing. 
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I’m going to start at the deepest depth that I’ve ever started a presentation. So I believe that 
for us to be able to successfully discuss these concepts in the end, I need to be very clear 
about what my background assumptions are. Then, I also believe I need to be clear 
about how I think those assumptions work or can be implemented. 


¢ What kind of theories and knowledge do I use to imagine a mechanism? 
¢ Then, I’m going to go into how I’m using all of that to think of virtual space. 


¢ And then, how we are using those ideas about virtual space to try to create AR and VR 


applications that begin to test some of those assumptions. 


So, the position part of the presentation. What are my assumptions? I want to propose first 
what's called ‘The Correspondence Theory of Truth’. This says that there is a reality out 
there, and its structure is homomorphic to our perceptions. What does this mean? It means 
that we don't know really what's out there, but we know that there is some correspondence 
between some sea of particles and radiation and whatever comes to our senses. In the history 
of human thought, this is a relatively new idea. And in everyday thinking and knowledge and 
culture, we still don't really take this seriously, as in, we still assume that we're seeing a chair, 
and the chair is brown, and we look outside the window and we see flowers and there's a 
certain colour. And that that reality is out there outside of ourselves. And even in reading a lot 
of the papers that are coming out of the scientific establishment, a lot of it is really not quite 
taking this proposition to heart that actually there is a huge gap between whatever that reality 
is and ourselves. And here I want to add a note that, actually, if you read words that are 
coming out from the computational branches of evolutionary theory, you will see that the 
correspondence theory of truth has refutations and it has fascinating mathematical refutations. 
So they're actually people out there who believe that there is no homomorphism between 
whatever reality would capital R is out there in our perceptions, that we might be completely 
imagining everything. But I will not go quite to that depth today. 

So there's something out there but there's a gap between that thing out there and 
ourselves, our perceptions. 

In practical terms, I like to make sense of this through what's called enaction theory. 
This was introduced by Varela and a few others in the 60s and 70s. I think in the book called 
The Embodied Mind (Varela, Thompson, Rosch, 2017) was published in 1990. And basically, 
this starts to deal with the fact that, this mapping between who we are and how we perceive 
the world in the world is really not tight at all. And it's not just that it's not tight, but we're 
continuously negotiating what this relationship is. And the reason why embodied 


cognition and the forum called inactive cognition is very important is because it triggered a 
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dialogue across science and culture that was about escaping what's called the Cartesian 
anxiety. So for many centuries, especially European-centric thinking was based on this idea 
that there is the subject and object, and they are two different things. That we have 
subjectivism, how things feel, and then there's objectivism, there is the world out there. And 
there are still a lot of struggles going on in a lot of fields to escape this Cartesian anxiety. It 
even goes into interesting discussions these days of what is consciousness and qualia and all 
of that and if we have free will, this is also about free will and all of that. My particular 
stance is to embrace Varela's inactive cognition and to stay there is no strict separation 
between who and what we are in the environment. We are defined by the environment and 
the environment defines us, and our entire organism is about negotiating this relationship. I 
know this is still a bit unclear, so I will just try to go a bit further into this. Basically, the 
proposition is that environments are shaped into significance, and these are quotes from the 
Embodied Mind by Varela. “Shaped into significance and intelligence shifts from being the 
capacity to solve a problem to the capacity to enter into a shared world of significance.” Or, 
“Cognition consists in the enactment or bringing forth of a world by a viable history of 
structural coupling.” So we become structurally coupled with the environment, and both our 
minds, our organism, and environment are adjusted through this structural coupling. And one 
interesting example that he gives in the book is of bees and flowers. We don't know if bees 
evolve the way they are because they are attracted to flowers who offer them nourishment, or 
the other way around, that flowers evolve beautiful colours because there were these 
creatures called bees that were attracted to them. Varela proposes that is neither or and that 
most likely both flowers and bees evolve together, to work together. So there was a common 
evolution because, from the point of view of the bee, the flower, and the environment, and 
from the point of view of the flower, the bee, and the environment. So each is both 
environment and subject from a different kind of perspective. And in that context, they 


evolved together through this structural coupling. 


This also ties back in terms of examples. To focus a little bit on examples now, if you're 
in Macy's papers from at the first conferences on cybernetics in the 50s, they were very 
concerned with research on frogs and I found that very interesting. So why were they so 
concerned with frogs? Because new research, at the time, showed that frogs cannot see large 
moving objects that... Actually, they can technically see but their brain just does not process 
large objects. So a frog is very good at catching small moving things like mosquitoes, but a 
frog will get run over by a truck. And it's not because the eyes of the frog cannot perceive the 
truck, is because the brain just doesn't process the truck. Large moving objects are not part of 
the frog's world. So that was actually very interesting and I think you can easily think of 


similarities or start to have questions going through your mind about what things out there, 
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that are very much in the environment and they very much exist, we might even see but just 
not perceive because they're just not part of how we deal with the world and how we interact 
with the world, they're outside the structural coupling that we have formed with the 
environment. And, although, this has been proved when it comes to frogs and many other 
kinds of organisms, we still have a hard time to imagine that, when we look out the 
window, there might be things out there which our cognitive system is just ignoring, 


perhaps, seeing but just ignoring, and I'll bring up some examples later in this regard. 


Another interesting thing is the ongoing research that's coming out about how the 
human eye is perceiving information. Here it turns out that, according to the latest studies, 
only about 20% of information that comes through the retina contributes to the image 
that we see to the image that a visual cortex forms. The other 80% is what's called top- 
down. So there's just other kinds of information happening in the organism that determines 
what we think we see out there, outside the window. Again, that number is now 80% and 
going up. And then, there's so much more out there in research in this sense. There's research 
that shows that if your hand is holding a cup of hot water, what you perceive from your other 
senses is different than when your hand is holding a cup of cold water. So just mind-blowing 
stuff that is just scratching the surface of this. Because we are still shaking off an 
intellectual culture of dualism, but also of this idea that we see what we see is what's really 
out there, many people still read about these things and catalogue them as illusions. And my 
work and my interests are about trying to understand to what is their limit and to what 
extent are they really illusions. And the more I work on this, and the more I read about this, 
the more I’m going down the rabbit hole of believing that they're not just illusions, they're 
probably correct. They're probably what the situation actually is. But why? Why do we think 
these are illusions? Why don't we perceive these variations? Or why is it so hard for us to 
even take these things into account? A lot has been written in what's called experimental 
phenomenology about the Necker cube. That cube that if you focus on it a little bit, it kind of 
shifts. And sometimes it seems like you're looking at it from the top-down, and sometimes 
from the bottom up. And again, everyone is cataloguing that as an illusion. It is not an 
illusion. And none of these things are illusions. But what's happening is, in the words of 
Merleau-Ponty, a French philosopher, very famous in the school of phenomenology says, 
“The world is pregnant with meaning.” So, we are born into a social world that fixes our 
perception to match a certain story. Our society tells us a story, and this story is very catchy. 
It's so catchy to the point where a lot of work and energy has to go into escaping that story. 
So our perceptions do not flip on us like the Necker cube. Because we are social animals and 
we share a story about what the world is. And what is that story? How powerful is that story? 
Well, it is that 80%. It is that, at least, 80% that is influencing the way we process the 
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information that comes from the retina, for example. 


The other word that I like in this context, also from Merleau-Ponty, is thickness. He 
says, “The world is also thick with meaning.” So it is very hard for us to cut to this thickness. 
And because most of the time we cannot, or it takes too much energy, we just buy into this 
idea that there is a fixed way to interpret information and that is the shared reality that we all 
live in. And, of course, a huge component of this, that he also goes into in his work is a bunch 
of norms that dictate not just what you should expect to see when you look outside the 
window, but also what's the appropriate way of looking out the window, and the appropriate 
way of behaving, the appropriate way of even thinking about these things, as in, cataloguing 
them as illusions that come with a certain baggage and so on. Okay. So what can we go 
deeper into the mechanism that starts to unpack how we interact with the perceptions and 
how they're fixed and what they're fixed by. And something that I found very striking when I 
was looking for the first studies and information on this topic, is the work of Lakoff and 
Johnson. They wrote a very famous book called Metaphors We Live By (Lakoff & Johnson, 
2008). They are cognitive neuroscientists interested in or working in the field of linguistics. 
And you're probably familiar with the work. The Metaphors We Live By was about how 
language has words like up, down, backwards, downwards, that are used in an abstract 
sense. And their conclusion was that metaphors are neural phenomena. They recruit sensory- 
motor interfaces for use in abstract thought. And this was just mind-blowing to me as I read 
it. I had to read it several times, not because I didn't understand what it meant the first time, 
but it was just so unbelievable. They're actually proposing that we take things that we learn 
by walking around in the environment, and then we use those structures to think. So in terms 
of a mechanism, explaining thoughts and perception I thought this was just absolutely mind- 
blowing. And there's actually a whole body of research that, both Lakoff and Johnson have 
done, together and separately, and other people, that are putting meat onto this theory. But 
again, because it's so unbelievable I feel like we're still struggling to really incorporate this 
into our intellectual culture. Varela also talks about how we lay down a path in walking. And 
a lot of people like this phrase, but many use it in a sense that's not literal. But read in the 
context of Lakoff and Johnson, I think, he might have actually meant it literally. As in, “Our 
thinking and our walking might not be different things.” 

Something that also points at a very interesting mechanism that deals with the 
muddiness of perception and thought is an article that came out in 2016, and it's about a very 
strange phrase called Homuncular Flexibility, the human ability to inhabit non-human 
avatars. And again, when this came out I had to read the title a few times because it was just 
so unbelievable. And it states basically that this thing, called Homuncular Flexibility posits, 


this theory posits that the homunculus is capable of adapting to novel bodies, in particular 
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bodies that have extra appendages. And that the recent advent of virtual reality technology, 
which can track physical human motions and display them on avatars, allows for the wholly 
new human experience of inhabiting distinctly non-human bodies. Ever since I read this, I 
started my own series of experiments in VR and I have discovered, to my surprise, that is 
actually extremely easy to, let's say, adapt to non-human bodies, to feel like you're truly 
embodying all sorts of things. I thought it would take much longer than it actually did. So, 
with technology like VR, these kinds of things are not even some super theoretical thing that 
can be achieved in a high-tech lab in some universities somewhere. It's actually in the hands 
of teenagers right now who are spending more and more hours a day on VR platforms, like 
VR chat. But I’m digressing a bit from the mechanism. So this is pointing again to a 
mechanism that is quite fascinating. Even things that we thought were fixed, like our 
identification with our body and our limbs, might really not be that fixed at all. And again, 
reading this, Lakoff and Johnson, metaphors that we recruit through sensory-motor interfaces 
are used in abstract thought, all sorts of things crossed my mind like, “Okay, so I’m 
inhabiting the octopus for a few hours. What kind of sensory-motor interface has that 
introduced into my brain and how will my abstract thoughts be changed by the fact that 
I’ve just spent half a day as an octopus?” Now, Merleau-Ponty and the traditional 
phenomenology and inactive cognition that I’ve started with, have been talking about things 
like this since the beginning and they all contain very precise examples of these mechanisms. 
For example, Merleau-Ponty has a famous story about how a man with a cane is actually 
using the cane as an extension of his body, because people who use canes, blind people who 
use canes, report feeling the tip of the cane touching the sidewalk. So they're actually very 
precise in that description if you read what they say about how they feel the graininess of the 
asphalt and the pavement. They really feel that they are there at the tip of that cane. So these 
mechanisms have been known, but I feel like now they are starting to be taken, quote- 
unquote, a little bit more seriously or their implications are starting to unfold much, much 
faster before us, because of technology like virtual reality. 

And here is something that, for me, it's also a mechanism, but it does not deal directly 
with perception, the movement of the body, and thoughts. It deals more with the sense of 
self. And I know that the sense of self is a very different topic than movement and 
environment, but it's going to come up later so I want to throw this in here. Foucault, the last 
book that was published about Foucault's writings is a series of lectures he gave called, 
Technologies of The Self. He never finished those lectures. He passed away. But this is what 
he describes as where he saw his work going, and what he would like to do next. What does 
he mean by technologies of the self’? He's very interested in what he calls the ‘emergence of 


a subject’. He's very interested in how people feel like they have a ‘self’ and an ‘I’. How they 
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describe that self and how that self changes. In this context, he's looking a lot at people like 
Rousseau and how Rousseau not only described the modern subject, but his writings actually 
contributed to what Foucault calls ‘The creation of the modern subject’. And this is important 
in the context of us dealing with, or having on our hands a piece of technology that allows 
people to spend half a day as an octopus. Foucault says for a long time ordinary individuality, 
the everyday individuality of everybody remained below the threshold of description, and 
then, people like Rousseau come in and start to describe how it feels to be human, and how it 
feels to be a subject of the modern state of France and so on. So, from now on, I will refer to 
this as subjectivity in the sense of, how does it feel to be a human self, a human individual, 
what could contribute to creating that particular form of how it feels to be you, and what 
could change how it feels to be you, and under what context does that change? And it's very 
interesting to me that Foucault himself uses the word technology, although in his writing he's 
not specifically looking at tech the way we think of technology right now. So just a quick 
summary, we're like halfway through. 


But I want to summarise a bit of what I’ve been trying to, kind of, do so far: 


¢ I’ve been trying to establish the fact that there is a gap between objective reality and our 
human world. 

¢ And my work is about trying to understand this gap a little bit better. 

¢ And the mechanism that, basically, connects us to the world, that does this structural 
coupling, in the words of Varela, is malleable. 


¢ And we are just starting to scratch the surface of what that means. 


But the establishment of this gap is the one thing that I want you to take away from the first 
part. I think I’m going to skip through this, but these are some of my favourite articles that 
I’ve been reading lately. They're all about how the things that we see might not, really, be 
about what's outside the window. They might be more about our own stories, and our own 
cognitive processes. It's that 80-plus percent that's about something else. And yet, we're 
talking about imagery, we're talking about what we think we see. 

This paper, in particular, maybe I’m just going to explain to you very quickly what this 
one is about, it's about this fascinating thing called ‘binocular rivalry’. These terms are, kind 
of, interesting sometimes: ‘binocular rivalry or ‘homuncular flexibility’. I’m very happy 
when scientists get so creative with naming these things. So, what is binocular rivalry? 
Basically, they did this experiment where they got a person in a room, and they showed that 


person either a face or a house, and then, they put some kind of glasses with a screen on that 


43 


person, some kind of VR glasses, that flashed for a fraction of a second either a house or a 
face. And what they found was that the brain decided to, quote-unquote, show the person, or 
the person then reported that they saw either a house or a face based on one they had seen 
previously, basically. So the pressing mechanism was like, Okay. I’m seeing a house, and I’m 
seeing a face. What should I give access to consciousness? Which one would be more 
relevant for the story of this individual? And the one that was, quote-unquote, shown to 
consciousness was, of course, the one that related to what the individual was shown at length 


before these flashes of images. 


So in this gap that we have established between reality, human beings, and our 
perception and thoughts, where and what are the strings, and can tech pull them? I think we 
have already answered this with things like, the homuncular flexibility and showing that we 
can inhabit an octopus and almost anything non-humanoid in VR. But I haven't seen any 
papers yet, maybe because this is just too crazy of a proposition, that takes the next step 
towards Lakoff and saying, “Okay. How does inhabiting that octopus then change the way 
you think? Change your thought process?” And, of course, there is no clear answer to that. 
The waters are very murky. The situation is incredibly complex. 

But the fact remains that, tech is starting to interfere with these things. 

And it's starting to get more and more powerful. 

And we are starting to see cognitive processes being altered. 

I believe we just don't have a choice but to start daring, proposing things and forming 
hypothesis, and going into the murky waters of the complexity of this whole thing as long as 
we want to work in tech. So how does this relate to virtual space? Because at the end of the 
day I’m an architect. And I’m reading these things, and what goes through my mind is the 
possibility to test these things by designing spaces. 

But before I go into a tentative framework that I’m using now, I want to start with what 
I call ‘Observations from Field Work’. So I spend a lot of time in VR. We develop a lot of 
VR applications in the office. I do a lot of events and talks in AltSpace and VR Chat. And I 
think it's important, before we dive into the theory, to also take into account just what are the 


stuff that I see out there that seems important. What is the bottom-up side of the work? 


The one thing that I find fascinating is what I call the Control+Z effect. This is a series 
of behaviours that I started to notice in myself, and sometimes in other people as well, that 
has to do with things you learn in VR, or in another kind of environment that, then cross over 
to physical reality and they reflect an inability of the brain to understand or to make a call 
between, “Okay. What are the rules of this reality that I’m in now and what are my behaviour 
allowances here versus my behaviour allowances in that other kind of reality?” And ’m 


calling this Control+Z because I first noticed it many years ago, and it was before VR, but 
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I’m seeing similar things coming out of VR. I want to say when I was an architect, I’m still 
an architect, but when I used to just do architecture every day without this whole tech stuff, I 
used to build a lot of cardboard models. But the workflow for my architecture projects was 
actually just many hours a day in a screen-based software product where I would just model 
things with the mouse and the keyboard, and then I would also, have in parallel, sometimes a 
cardboard model running of the same thing, so sometimes I would make decisions in the 
screen-based software, and sometimes in the cardboard model. And on several occasions, late 
at night, when I was tired, so my brain was kind of struggling a little bit. While working on 
the cardboard model and making a mistake, my left hand would immediately make this 
twitching movement, and my fingers on my left hand would position themselves in the 
Control+Z position of the keyboard while I was working on a cardboard model. And I would 
always be kind of surprised, and then, of course, similarly realize what had happened and 
catch myself in the act and shamefully, a little bit, put my left hand down, “Okay. There is no 
Control+Z.” But what was happening was, basically, my brain was, kind of, deep into this 
screen-based computer software where there is a ledger that records all the actions that you 
do in that environment in time. And you do Control+Z and then you go back one step in that 
ledger. So my brain had gotten used to the idea that, that environment, quote-unquote, and 
reality can also go backward. And then, of course, in physical reality the hour of time does 
not go backward. So that's the first observation. 

Then, I’m seeing a lot of emerging phenomena in virtual worlds. I’m seeing people 
discover new possibilities for being, for interacting, crazy things happening in VR Chat, if 
you're not familiar with that platform, I highly recommend it. I think it's by far the most 
advanced VR interaction you'll see, and worlds being developed, and forms of community 
building, and community life intermediated by this technology. All of that is happening in VR 
Chat. And they're years, years, years ahead from any other kind of experience, or game, or 
anything else that I’ve been seeing. So I’m seeing signs that there are emerging social 
dynamics and mechanisms for negotiating meaning in these collective groups and 
interactions that are extremely interesting. 

This is also a bit of a topic for another day, but I feel like it's so important that I cannot 
not mention it. We're slowly but surely not the only intelligent agents anymore. We 
interact with bots on Twitter every day and we don't even know that they're bots sometimes. 
And people are experimenting with introducing all sorts of Al-driven agents into virtual 
worlds. We have Unreal and Unity putting out their extremely realistic-looking avatars that 
are AR driven and so on. So we're not really at the point where we go to VR Chat, my 
favourite platform, and we're not sure that the other person is human or not. But I think, well, 


I don't know, if we're not already there, we will be there pretty soon. So there's a significant 
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layer of complexity that's being added right now on top of this already complex and messy 
situation, by the introduction of non-human cognitive systems. 

All right, so what is the proposition for what is virtual space? This is how I think about 
it. Anew environment is basically a system you're trying to solve. It's a little bit like a 
game. So this is the structural coupling of Varela. You go into a game, you go into a new 
building, you go to a new country to visit, you've just landed at the airport, the first thing you 
do is, you're trying to figure it out. You're trying to understand where you are and which way 
you go. Are there any things that are strange? Your brain is turning fast to establish, as soon 
as possible, this structural coupling with the environment, that gives you control over the 
environment and understanding. 

But I want to argue, in that process, you're not just dealing with this foreign 
environment, you're actually also encountering the system that is you. You're also dealing, 
and discovering your own cognitive processes that are engaging with the environment in 
attempting to couple. So roughly put, designing the environment is designing the subject that 
interacts with it. So how would an approach to space making look like if we just assumed, in 
the light of all of this talk about cognitive neuroscience, that the environment and the person 
are the same thing? That, somehow, they're so tightly connected we cannot disconnect them. 
It's like the bee and the flower. 

If we were to pursue this kind of methodology, what would our tools be? Where 
would we even start? And I can only tell you how I’ve started doing it. I’m basically doing 
the best that I can to form hypotheses that have to do with knowledge that I’m taking from 
these papers, and knowledge that I’m taking from my own experiences and introspection. 

One of the mechanisms that I’m very interested in now, and I will show you how we 
use that in one of our projects is the fact that, unlike other kinds of screen-based software or 
interfaces, screen-based interfaces that only address or mostly address our visual cortex, VR 
throws in the ability to control or encourage behaviour that activates the motor cortex. And 
this is an absolute game-changer because, as a lot of these papers reveal, it is the organism's 
attempt to integrate sometimes, perhaps, conflicting information that comes from the motor 
cortex and the visual cortex, that it's one of the most important paths that we have in trying to 
understand more complex cognitive paths. 

One way is to try to understand this relationship, and then to try to use VR to test 
things. So what if the eye sees something, and then the body does that, what happens next? 
Can you always predict what the person there will do? You can if you only show them and 
make them do what they would see or do in physical reality. But the moment you depart from 
that, the moment they either see something else and do something they would do in physical 


reality or the other way around, very interesting things, very quickly start to happen. Now, to 
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what end? I think this is something that will have a different answer for every developer or 
every company. I think this is, primarily for me, a methodology that I’m only able to pursue 
and explore using VR and AR. This is not something that's possible for me with traditional 
forms of architecture. That's primarily the reason why, as an architect, 1am in AR and VR 
and not just in traditional architecture. To what end? For me, the answer is that there are 
many answers, but one today is that I’m interested in new ways of thinking, and new ways of 
subjectivity. So that's why I introduced that slide earlier about Foucault and subjectivity. ’M 
INTERESTED IN NEW FORMS OF BEING HUMAN. And I think that can be pursued 
through this kind of methodology, but we'll see how things go in AR and VR. I think, new 
forms of subjectivity can also be pursued through traditional architecture, but there are many 


reasons why that is a little bit slow. 


Okay. And now the last part of the presentation is the fun part. This is where later you 
can tell me, “Hey Andreea. The things you said, and the things you did, or just the way things 


turned out do not quite match.” But I would love to hear those kinds of questions. 


On the following pages: Implementation 
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Say It 


How do signs and 
shapes acquire 
meaning in collective 
discourse 


This is an older project, but I think it's very relevant in this context, so I decided to start with 
it. This is a, let's call it art project, it's called Say It. Basically, I designed these different 
shapes, they're in wax here because I was planning on pouring them in bronze. I never got to 
pour them in bronze and integrate these RFID tags into them. But basically, this is based on a 
story from Gulliver’s Travels. Gulliver goes to Lilliput. That's the country with the little 
people. And he runs into these Lilliputians that cannot speak in words, they speak with 
objects. They carry on their back a big bag with an object, that's a sample object of all the 
objects that they need to communicate. So if they want to tell you something about spoons, 
they will go into their bag and pull out a spoon and show it to you, and then you're supposed 
to like, quote-unquote, read that they mean to say spoon. So this intersection between 
language and objects, or objects as language, and then, the many complications that result 
when trying to use objects as language, because you don't have syntax, was something I 
became very interested in. So what is the syntax if you just have the objects? How does that 
arise? So, the idea with this project was to have two people and then give them a bag of these 
objects, and these are somewhere in between letters and objects. And to design ways in which 
this could maybe give some sort of feedback. But to observe how fast, or to what extent, or in 
what direction people start to use these to communicate. The people are not allowed to talk to 
each other, of course, so they're given something they're meant to communicate to each other 
and only have these objects. And then, they're given an hour to try to use these things to 


communicate, and basically, they have to negotiate meaning for these abstract shapes. 
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BEREIT? — 


GeheimnisGraberel: Ein — 


AR triggered OBE 


This is an AR game that we have developed for a museum. And here we used one of these 
approaches that I mentioned earlier. We hypothesise a certain reaction that would happen if 
we present the visual cortex with conflicting information from what the motor cortex is 
reporting to the central nervous system. And it worked. We were able to trick people into 
believing that their body is floating upward. About 20 meters. So we basically trigger the 
mild out-of-body experience. This is mild, it's something quite nice, it's a game that happens 
outdoors, it's triggered by GPS coordinates and you're basically exploring a story of the 
German [indistinct] in the south of Germany. It's very integrated with a story. It's a very mild 
thing. It's not scary at all. But we were surprised ourselves that we were able to use some of 


these theories to make something like this that actually, quote-unquote, works. 
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NUMENA 
Lane 
Metaphor as 
strategy. Semantic 


vs spatial memory 
Tags in space 


This is a three-dimensional menu. What you're looking at here is, basically, a folder with 
files. It's something that, from the technical knowledge that we have today, it's something 
very basic. Something a programming student will understand everything about in the first 
hour. But we wanted to see how we can take a folder with files and make that a three- 
dimensional experience. So we went very literal about it. We used what is called the 
metaphor approach to UI, UX, and interfaces, but with a bit of a twist. So you are in an 
elevator where you can go up and down to infinity. And in each one of these TV slots, you 
can save one of your files, that you produced in this application that we're working on. You 
can save it in here, and you can then rearrange them, because we're working on putting smart 
tags on them. So it's kind of like creating a map, but then, you can reorganise them so that 
they form a different kind of map. And what's even more interesting it's, we also tested 
another thing. You can go in, on this chair, and pull a file out of this slot next to this strange 
TV screen and throw it down into the abyss. It's like a big VHS tape that you kick outside of 
this chair and you can look down and see it drop. We're very interested in understanding how 
people react when they have to interact with abstract things like files as if they were physical 
objects they can throw. And this is part of a much more complex exploration that we're 


pursuing. This is part of the same application. 
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NUMENA 
Lane 


This is the kind of environment you can make that you then save on the screen. And the one 
thing that I want to point out here is that, you basically see the scene two times. What you're 
seeing here is that, you are in this roof that's shown to you at one to one scale, and you also 
have a mini version of that roof. So you're simultaneously perceiving, quote-unquote, this 
fake reality inside of your headset two times. And we're experimenting with all sorts of 
interactions in here, because you also exist in here two times. You exist at your perceived 
one-to-one scale. And what we call “mini-me” is also in here. So there's mini you in there 
that you can also interact with. So we're seeing very interesting things happening because, of 
course, this environment, where everything is twice and there's a mini you that you can do 
things to, it's a very different logic of the universe than what we are used to having in 


physical reality. 


at 


This is a Borgesian Infinite Library based on a Penrose tile pattern. We made this kind of 
for fun to explore the limit like the psychological limits of environments. This is actually a 
VR environment, but it's a bit much so when you go in, your mind starts to lose it a little bit. 
But we just wanted to make an environment where we observe, at what point is an 
environment too much, and what exactly are the psychological effects that you start to 
experience in the first person when that environment becomes too much. And why is it too 
much? Is it the repetition? Is it the modularity? What exactly makes triggers those 
psychological effects? 
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™ GravityX 


And this is my last slide. This is a game that we're working on, also highly experimental, 
where we're putting a lot of these things that we're thinking, and reading about, and 
exploring. We're collecting all of this into what we call a VR testing environment that is 
called GravityX. And the motto for this is, the first line from John, but with a bit of a change. 
So it goes, “In the beginning there was space, and the space was with God, and the space 
was God.” So we basically replaced the word, “Word” with space in the first line from John. 
All right that was it. Thank you for bearing with me through this. 
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O&A 


https://youtu. be/4 YO-iCUHdog?t=3 864 


Frode Hegland: It was an absolute pleasure. Very, very grateful. I mean, obviously, lots of 
questions and dialogue now, and amazing. My initial observations, kind of, to you and to the 
group. First of all, thank you. And secondly, I was asked a while ago about, “Do I think the 
future is going up, improving? Or going worse?” And my answer was, “It seems to be 
diverging. Getting much better and much worse”. You're in Germany now, right? So we're 
dealing with a full-on war in Europe. We're dealing with horrible things in other parts of the 
world. And then, we have this. When I defended my viva to Claus and Nick about two weeks 
ago, they very rightly questioned some of my language use around mental capacities. And my 
defence to them was, “We just don't know enough to use hard language”. So Claus, if you 
don't mind taking the first half of this presentation mentally into my thesis, that would be 
great. What I’m trying to say with that is, if our species is to survive, we have to evolve. And 
we're the only species known who has a chance to have a say in our own evolution. So I think 
that what you have shown today is foundationally important. It was just really beautiful. We 
have to take this very seriously. In our group here, we call ourselves the Future of Text Lab. 
But we have decided that what we mean by Text is almost anything. It used to be very 
narrow, but because of VR, we're doing something else. And just two more comments before 
I open up the virtual floor here. One of them is: I believe that the most powerful thing human 
beings have is imagination. And imagination has an enemy, truth. A teacher, when I was in 
university, many years ago said, “Truth kills creativity. Because when something is 
something, it is something and you're not going to look at it in a different way”. We saw that 
with the normal, traditional desktop computing, it basically became word processing, email, 
web, and a few other things. A lot of the early stuff isn't there. When we today, in our 
community, try to make more powerful things, people say, “Huh. But that's not a word 
processor’’. Or, “Uh. That's not that”. Because imagination has been killed by truth. It is 
something. A little thing that I read on New Scientist, I think two days ago, in our bodies we 
have this thing called fascia, which is a connective tissue that goes around all our organs. I’m 
mentioning it for two reasons. First of all: it is kind of like an internet for our body that's not 
our central nervous system. But until 2019 it was just thrown away. If you're doing a 
dissection, or if you're cooking a beef dinner, you would just get rid of this stuff. Because we 
didn't have the ability to investigate it. And again, 2019, nobody had looked at it before. And 
now we're realising that it has about as many nerve cells, roughly 250 million, as our skin. 


When you are looking at the way that our brain connects with the world, what I really liked 
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about the way you do it, you are clearly very intelligent, but you're also very humble. Clearly 
we have evolved with our environment, but the implications of what that means is extremely 
hard for us, humans, to fathom, I think. So, I just wanted to thank you very, very much for 
having the guts to look at this most foundational thing of what is to be human. And for 


us to together try to use virtual reality type things to examine how that may change. 


Andreea Ion Cojocaru: Yes, thank you so much for saying that. Well, I think I have the guts 
to talk about these things because I’m an architect. 


Bob Horn: I’m so excited by this presentation. It's just so delightful. George Lakoff was a 
friend of mine and colleague. I audited his course over in Berkeley. I wrote the obituary for 
Varela, for the World Academy of Art and Science. The whole framework in which you 
enmeshed us in now is wonderful, and it really excites me now to get into virtual reality. 
I’m among the older people here in this group and I’ve resisted. Gulliver’s Travels metaphor 
was wonderful. I have a collection, one of the things I do is put words and images together. 
Visual spaces. As you can see behind me. Mostly I do it into two-dimensional murals that are 
12 feet long and so forth. I actually work with the International Task Forces on this. The one 
behind me is the one I did on the avian flu 15 years ago. On what could have been the worst 
pandemic. And so, anyway in looking into into just the Gulliver thing. I mean, that I want to 
get off my mind. I had forgotten all about this bag of stuff. I have a bag of objects which are 
arrows. Which I use in these murals. I have a bag of 200 arrows. Different kinds of arrows, 
that have different kinds of meanings, that I would like to throw out there and give to you and 
see what you do with them, and see what you do with them in in virtual reality. So, anyway, 
I’m just filled with exciting possibilities after this. I don't want to occupy any more time, but 
thank you very much. It was wonderful. 


https://youtu.be/4 Y O-iCUHdog?t=4317 


Brandel Zachernuk: Thank you. This is super exciting. And your comment on the, sort of, 
the homuncular flexibility and, sort of, hinting at neuroplasticity is something that I’ve 
definitely observed in my work. I was one of the responsible for some of the launch titles for 
Leap Motion. One of the things that were really fascinating for me there was having the 
number of degrees of freedom that one has there, and being able to just turn those things into 
whatever you wanted. And after a while, the contortions that one's hands were undertaking, 
completely disappeared. And the more simple of which was just tilting a hand, but then, 


amplifying that three to four times. Most people didn't realise that this angle wasn't that 
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angle. They completely thought that their hand was down, despite the fact that that would 
have been anatomically impossible. So I think that we have an enormous range of 
opportunities available to us once we have the ability to, kind of, recruit more of our stuff. 
One of the first things that I wanted to talk about, or ask you about is; You were pretty 
disparaging of the term "Illusion," which I’m in agreement with. It reminds me a lot of 
Gerard [indistinct]’s frustration with people talking about cognitive bias and the sort of 
embodied situated cognition kind of things you're talking about also, prioritise cognition for a 
reason. So have you come across or what is your take on cognitive bias and how it relates to 
this, as well? 

Andreea Ion Cojocaru: Well, most of the things I’ve encountered that were referred to as 
cognitive bias, where bias, with respect to some kind of main understanding of cognition, but 
we do not agree on what the main understanding of cognition is. So I don't know from what 
point of view do you think that that particular thing is biased. So I don't find those 
conversations particularly useful, or the term itself, from the perspective of my interests. 
Because I don't think we have that common ground or understanding that would allow 


us to meaningfully talk about bias. 


Ken Perlin: Everything you're saying is absolutely wonderful and resonates very strongly. 
And it also, in support of this, I’m thinking that there's this phenomenon that, when 
something becomes normal, we tend to forget that there was a time when it wasn't normal. So 
everyone here has had the experience of an automobile being an extension of our body. And 
we all read a book, which is an object that kind of didn't exist at some point. Even the fact 
that we wear shoes now when there was a time when people didn't wear shoes, the whole 
world would have seemed very strange. And obviously, phones and all these things. So it 
seems to me what you're talking about is kind of the next phase, or actually putting some 
rigour behind, a phenomenon that is because we are the creatures of language, so, therefore, 
we live in this world where I say the word ‘elephant’, you've got an elephant in your head. 
And that happened a hundred thousand years ago. We're kind of catching up in some sense to 
understanding what we do as a species. And I think I agree with you completely that, because 
of the more radical vestibular nature of, “I put on a VR headset, and now I start having these 
new kinds of novel mappings”. But, on the other hand, the language of cinema is something 
that might not have made any sense to someone before we all learned how to watch movies, 
and that's a completely crazy mapping, if you were not used to it, that radical point of view 
changes from moment to moment, but yet doesn't drive us crazy. So I feel like, not only is 
what you're saying make a tremendous amount of sense, but it's also making sense of 
things that happened long before we even had computers. And that's kind of what we do 


56 


in a way, we just didn't kind of acknowledge it yet. And I wonder, what do you think about 
that? 

Andreea Ion Cojocaru: I think we're social creatures. So sharing a reality is how we 
survive. It's the kind of organism that we are. So it's important that we can share a reality, and 
the reality that we share cannot be the actual reality. It's just not. So we share a story about 
that reality. And it takes society to change the story. Individual people cannot change the 
story at a level that's profound or meaningful enough at all. There are these lonely people that 
sometimes can become important, and we call them innovators, when everything is good we 
call them a pain in the ass. I think now is a particularly difficult time in which we happen to 
need innovators. I think now things are not looking good at all in terms of where society is 
going and what we're doing to the planet. So I think there's a particular urgency to call the 
people that can shake up the story. That's also a bit the reason why I introduced the talk about 
subjectivity. I believe that there are two reasons why I go into these things with VR. One is 
because I personally believe this is a path and a methodology that gives us the most ability to 
understand what the technology can do. But I also think the promise of a change, in the 
subjectivity of a change in the story, collective story, of a change in how it feels to be human 
is appealing to me, because we are, at a point, where we really need that right now and we 
can't afford to wait. So there are two slightly different reasons why I chose to kind of go 
down this path. And, yes. I think all of this has happened in the past. I think the collective 
story controls the narrative of everything. That's why, for me, the moment VR will reach 
mass market is actually very important, because, right now, we're still talking about this 
technology being at the fringes. We have what? Half a million people? A million people in 
VR Chat? But I think the numbers are much less in terms of concurrent users. But where are 
we taking things if half of our teenagers start spending half a day as an Octopus, how do we 
make sense of that, and how do we take this tech to a point where we... It's like, I think that if 
we continue to avoid a serious discussion on these mechanisms and methodology for XR 
developers, we will fail to have a good grasp on this technology. It's a hard conversation 
because a lot of people, as I said, either believe that these things are illusions or do not think 
is part of their discipline to go into this discussion. My position is, you just don't have a 
choice. We just have to go this path. Or at least have a conversation and debate 
methodologies. Because we will be in a situation where, on one hand the whole planet is 
going down the drain, and on the other hand we have to put half of our teenagers in some 
mental institution because they spend their days as an octopus. So this is putting it extremely 
bluntly. I should mince my words, but sometimes I get this sense of urgency coming from 
these two directions. And the best I can do, with my ability to think through things, is to go as 
deep as I did today and try to ask these difficult, unanswerable questions, to try to prevent, 
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perhaps, or contribute to the prevention of these two big dangers that I’m seeing. 


Ken Perlin: Thank you, yeah. It will come a day when the people who get put into 


institutions are the ones who refuse to learn how to be an octopus. 


Mark Anderson: | love this. Interesting enough, actually, it was interesting the bit about 
homunculus, because that actually, my understanding sort of came at a completely different 
angle, because I came across it in V. S. Ramachandran's book, Phantoms in The Brain, back 
in the late 80s, where this was about to do with neurological people with damage and how 
they were adapting their bodies. But, of course, it's blindingly stating the obvious, to me it 
says that this would map across, why would it not? Because just if you can wrap your mind 
around mapping your mind away from a limb you no longer have, putting a couple of extra 
octopus arms on isn't such a big stretch. I just come back to a couple of things that it's 
interesting to sort of getting your thoughts on a bit more. I was listening to your thing about 
the Command+Z and I was just wondering, it was hard to phrase this in a way that doesn't 
sound glass half empty, which isn't where I come from, but so when we bring these things 
back, I suppose the answer is we don't know whether we bring back good things or bad things 
because, in a sense, we can train ourselves to do things we do normally for not particularly 
societally good reasons. We train people to do things very well. And then we have problems 
teaching them to not do that. So I’m wondering if there's another interesting element in this 
as we explore it. On the one hand, potentially the gain, even the things, going back to my 
opening point about the neuroscience people at San Diego trying to mend broken bodies and 
things. But just being able to effectively work through a different set of control 
mechanisms is really interesting. So I don't know if you have any thoughts on that. And the 
other thing that I was interested in, when you mentioned sort of the 80/20 thing back you 
were also saying effectively we're not using, or we don't know how we're using 80% of 
our neurological inputs. Is it that we don't know what it's doing or we just think it's not 
being used? 

Andreea Ion Cojocaru: Yeah. Oh, I can clarify that. The first example of this that I’ve 
looked at, is actually Varela's own research. He was studying vision. And he talks about this 
in The Embodied Mind in 1990. He talks about how, basically, 20%... So the information is 
entering through the retina, the optical nerve. And the visual cortex is forming the image. So 
that's what our consciousness perceives as it's out the window. And Varela concluded from his 
own studies on vision that only 20% of the information that's coming through the optical 
nerve is used by the visual cortex. And there's very recent research, a few months ago, that is 
reinforcing that about various parts of the brain. So 20% is like, quote-unquote, actual. But 


actually, the thing is, the percentage, in the beginning Varela was not really believed, and 
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there was a lot of pushback on that. They were like, “There's no way this is true’. I've 
recently listened to a podcast by a neuroscientist saying amazing, completely shocking things 
are coming out of research right now showing that 80% or more is what's called top-down 
influences. And she sounded completely like, “Well. But this is science, so we must believe 
it. But we still can't really, or really want to believe it. And it looks like there could be more 
than 80%”. And she was kind of shaking. Her voice was shaking as she was saying that. And 
I was like, Well, Varela said this 30 years ago. So there's some degree of homomorphism, but 
again, if you listen to other people, there's no homomorphism, there's some degree of 
homomorphism between the environment. It is that 20% or less, the rest we're making it 
up. We're making it up. But it's a collective making it up. 


Peter Wasilko: I was wondering if you had any thoughts about the use of forced perspective 
and other optical illusions in real-world architecture in order to create a more 
immersive environment? 

Andreea Ion Cojocaru: I think, in the physical world, we are experimenting with AR in 
creating illusions. I don't know if that's what you mean. So my example of the AR app where 
we create this out of body experience was a little bit like that. But for me, it's very much 
connected with what are we trying to achieve. And for our work, it's not immersion. I’m not 
very interested in immersion for its own sake. It's like, what does that mean? Does it mean 
you really believe that you're in VR? I don't know if that's so relevant for my interest. We 
create illusions but only because we want to achieve a certain feeling, or emotion, or 
cognitive process, or trigger a certain thought process. So the illusion has to be connected 
to that by itself just being in an environment, and thinking it's another kind of environment, 
or if thinking, or having the illusion that is bigger, or smaller, or just different on its own, 
without part of the largest strategy, is not something that we would typically pursue. I don't 
know if this answer your question. 

Peter Wasilko: Yeah, pretty much. I was thinking of trying to design environments to 


achieve certain emotional cognitive effects. So I think we're running in the same direction. 


Claus Atzenbeck: Yeah. First of all, thanks for this talk. I have three quick questions, I 
guess. So you showed one project. It was this elevator, basically, which you can use to go to 
some TV screens. Can you Say a little bit about the limitations we may face in a virtual 
3D world? For example, if I imagine that I have some zooming factors implemented that the 
user could zoom in to up to infinity, basically. This would change the perception of the room. 
So I would become smaller, and smaller, and smaller, and the space would just become 


bigger and bigger so I could, actually, have different angles. So is this something the human 
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could still work with? Or for example, what about rooms which are of contradicting 
dimensions? I imagine this Harry Potter tent, for example, which is larger inside than outside. 
Is this something a human can actually deal with? Could a human, actually, create a mental 
model of, since this cannot happen in the real world? This was the first question. 

The second one is a general question about vision space, VR, I mean, this is all about 
visuals. This is just one channel, basically, we look at. Did you think about, well, first of all, 
why did you pick that and not other channels which would target other senses? What do you 
think about multi-modality, for example? Using different senses? And also, what would be 
the potential, basically? When you said this Control+Z thing, I thought about the muscle 
memory I have for typing a password, for example. When I actually look at the keyboard, it 
becomes harder for me to type in the password. And if I see a keyboard which has a slightly 
different layout, possibly two keys would be exchanged, like the German keyboard and the 
U.S., American keyboard, it becomes almost impossible to type this password fast enough, 
because I’m kind of disturbed by the visuals. So wouldn't it make sense to actually ignore the 
visuals for some projects, at least, just thinking about the other senses, basically? 

And the last question is more of a general nature. Do you think it's really beneficial to try 
to mimic the real world within the computer? Like a 3D world which almost feels like 
being in the real world? Or do you think we should focus on more abstract information 
systems which may be more efficient, for example, than using an elevator going up and 
down? 

Andreea Ion Cojocaru: Yeah, thank you for that. I think one and three are connected. One 
and three are about the elevator. The first question was; could it be too much for us to deal 
with these infinite spaces and this shrinking and expansion of our perception of the 
body because it's so drastically different? Up to a point, we can definitely do it. Just like 
the octopus. I do think we can do it. We will hit boundaries and borders, and I’m fascinated 
by that. So part of our more experimental work is to see where those boundaries are, and 
what does that mean. Because, yes, we have adapted for quite a while through the physical 
Reality with, capital R, whatever cloud of particles and radiation that is for quite a while, 
right? But if the people that do not believe in homomorphism are right, and mathematically 
so far they look like they're right, we actually have no structural coupling with what is out 
there. We completely make up the collective reality. But again, I’m going into speculation. 
Since I’m like not a scientist, I try not to speculate in public. And when I speak in public, I 
just focus on the papers and keep the speculation to my interpretation of the papers. Going in 
this direction would mean going into papers that are not commonly accepted as science. So 
it's a big parenthesis. I believe, assuming we have homomorphism structural coupling with 
Reality with capital R, I think we will hit boundaries. I think VR can quickly put us in 
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environments that we can't deal with and will feel uncomfortable. I’m interested in exploring 
that boundary and have... I don't want to go beyond boundaries, I have no interest in making 
anyone feel uncomfortable. But I feel like we don't really know what the boundary is. So 
we're talking about what we think the boundary might be, without actually having a good 


understanding of where that is. 


Then, the third question was related to the chair. So I would argue that that chair is like 
nothing you would ever experience in reality. We're taking something that is a little bit 
familiar to you, which is a chair and a joystick that moves the chair up and down, but the 
experience and the situation are drastically different than anything you would do in reality. 
Because you cannot take a chair to infinity in reality. So what we were doing in that 
environment, people say skeuomorphic, I’m like, "What is skeuomorphic about driving a 
chair to infinity?" So what we were doing is, we had some variables, some things that were 
controlled. We couldn't have variables everywhere. We couldn't have variables on the infinite 
wall, and variables on the chair and what's around you, because it would have been too much. 
So we made the chair and the control skeuomorphic, quote-unquote, so we can experiment 
with the other stuff. And the fascinating thing was that, basically, that environment is just a 
folder with files. But just by doing this, it's stupid, the whole thing is on the infinite elevator, 
and the infinite wall, on a basic level is the dumbest thing, but all of a sudden, people started 
to get exactly the same ideas that you just got with like, "Oh. What if I go to infinity? What if 
I start to have the feeling that I’m shrinking or expanding?" And you do. You do start to feel 
like you're shrinking and expanding and you're losing your mind. People started to think, 
“Oh. I could have infinite scenes”. This is like, they started to ask us, “Is this the metaverse? 
Oh, my God! The possibilities of seeing all of my files in here”. And people got excited about 
something that they already have. They already have that in a folder. You could almost have, 
well, not infinite, but you could have more files than you would ever want in a folder running 
on a PC. But their minds were not going, and exploring, and feeling excitement about those 
possibilities. So it was interesting how, just by changing the format, like spatialising 
something you already have, just open up this completely different perspective. So, yeah. We 
call that our most spatial menu yet, because that's basically a menu. I think there is 
tremendous potential in this very simple, almost dumb, shift from screen-based 2D interfaces 
to 3D. It's dumb but for some reason no one is doing it. For some reason like, I posted this 
stupid elevator and some people were like, "Andreea, this is stupid. What the hell is this? 
Why are you doing skeuomorphism?" Because I’m known for these ideas, and known for 
hating skeuomorphism. And everyone saw my elevator was skeuomorphism and I’m like, 
“No, no, no. That's really not what we're doing”. And every single VR application out there 
opens a 2D menu on your controller and you push buttons. And it has like 2D 
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information. So they're still browsing files and information in VR on a little 2D screen. So 
this elevator was our attempt to put out there a truly spatial file browser. And the extent 
to which it triggered this change in perspective over who you are, what do these files 
represent, who you are in relationship to them, what is the possibility, was really striking. We 
didn't really expect that. We almost did it as a joke. We were almost like, “Why don't we 
model this like 60s soviet-looking elevator and then, have an infinite wall and see what 
happens”. The idea with the infinite wall also came from like, I have a few pet peeves: 

One is like, homomorphic avatars, which I hate. 

The other one is the infinite horizontal plane that all the VR applications have. 

Why in the world do we have this infinite horizontal plane in VR? 

So we wanted to make an infinite vertical plane in VR. Muscle memory, yes. So the 
reason why we're focusing on visuals is because that's what we've been focusing on. But in 
the game that I mentioned, we have an entire part of the game which is called, The Dark 
Level. So what we're doing in the dark level is exactly what you said, which is we're 
exploring sound and space. You don't see anything. So basically, the VR headset is just 
something to cover your eyes and to get sound into your ears. That's something brand new 
that we're embarking in, because I agree with you, everything that I talk about is not 
necessarily specific to visuals, it just happens that we're just now starting to do space and 


sound, as opposed to space and visuals. 


Claus Atzenbeck: Just one more question on what you just said. Do you think this infinity 
virtual 3D environment is something that people like because it's something new but 
you're not solving a particular problem? Because I can imagine that we have a plain 
zoomable user interface like Jef Raskin did something like that, which you can zoom in and 
check your files on an infinite 2D space on canvas, basically, on the screen. So it's just 
because it's something new and people are happy to use that because it's new? So it's like a 
game? That's gamification, basically? 

Andreea Ion Cojocaru: There are two things we're pursuing with that. 

One is spatial memory as opposed to semantic memory. There are studies that show 
that spatial memory is more efficient than semantic memory. In other words, you're more 
likely to remember where you put something than how you named it. So we're interested in 
where people put things. And we don't want people to put something somewhere, this object 
that is their file, with the mouse. We want people to physically move their bodies to put that 
something there. So we're taking the file, which is an abstract thing, we're embodying it into 
an object in VR, and we're making people, literally, take it with this forklift, because we're 


just being stupid right now, with this forklift and literally putting it somewhere else. So that 
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kind of testing of spatial versus semantic memory, I think, can only be done in this context. 


And I don't know of any other project that's doing it. 


And the second thing is, yeah, just this pure idea of interacting with abstract entities 
as if they were embodied objects, and being able to apply physical movements of the body, 
and moving the body through space to interact with these abstract objects. So that's kind of 
clashing together Lakoff with all of these other theories. It's like, you're learning how to 
manipulate abstract thoughts, by learning mechanisms from how the body moves to space but 


in a perverted kind of way, VR allows us to smash the two together. 


So we are, and we are just observing how it happens. So, no. At a conceptual level, we 
would love for people to have fun, but it is these two things that we are interested in learning 


more about. We have not just made it so people think it's just cool to go up and down. 


Frode Hegland: I’m going to go all the way back to that 80% stuff. That, of course, in a very 
real sense doesn't mean anything. I’m sitting outside now and there are our trees, and birds, 
and everything. And we have to talk, of course, about affordances. What these things are to 
me, which is interesting. I can see that there's grass over there. There's no chance and no 
usefulness for me to know exactly how many blades of grass, exactly what angle they are, 
exactly what colour level they are, etc. That is not useful information for me. So obviously, 
the 80% stuff is all about where in our system, information gets filtered. And how it's used. 
There are, of course, different levels of this, and the reason I wanted to discuss this point is, 
in the physical world, if there is a fox or something that may come gnarling up at me, then a 
certain type of shadow has information that otherwise wouldn't have information for me. And 
it'll be very interesting to see when we start designing our environments in virtual 
reality, how we can choose to, more intelligently say, “This stuff is meant to be here 
because if it wasn't here, you would wonder why it's missing”. Like a wall. You know you 
don't need a wall in VR. But otherwise, it would feel unbounded, literally. And here's another 
piece of information about this wall, which has actual meaning to you. So I’m wondering if 
you have any reflections on, let's call it hyper surrealist worlds, where you look out the 
window and you can choose to see the weather tomorrow. Some of it's kind of real and fancy, 
some of it is just completely insane. But that thing where some information is meant to be 
there, otherwise, you'd miss it. Other information has actual meaning. Thank you. 

Andreea Ion Cojocaru: Yeah, thank you for this question. I’m going to say some things now 
that I allow myself to say in public because I am an architect and not a cognitive scientist, so 
I’m not going to risk my reputation. But the reason why the 80% is meaningful to me is, 
because it means the 80% can be changed. The 80% is the story. So, again, this is kind of 


very out there statement, but I’m more interested in figuring out, rather than changing the 
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environment and designing super interesting environments, and putting people in there. I’m 
very interested in pursuing what these research studies are implying and seeing to what extent 
the story can change what you see. Because the “over 80%” is the story, so if we change the 
story, you will not see grass anymore. Just like the way the frog cannot see a truck. Again I 
don't mean this quite so literally, but on the other hand, I do. On the other hand is the study 
that shows that if you're holding a glass with hot water, you hear different things than when 
you hold the glass with cold water. So the evidence is on the wall, but we are really scared of 
going into the implications of this. And the cognitive scientists do not risk their reputation. 
Some do and talk about things, but they're not exactly considered mainstream. So it is there. I 
mean, the study is there. 

Frode Hegland: Oh, yeah. And I think that's phenomenally useful, but another half of this is 
the issue of... I had a friend who was obsessed with cars. He would know everything. So we'd 
be walking down the road and he would see, at night, a taillight from behind, at an angle, and 
he could tell me who designs the wheels of that car. So what he saw, what was information to 
him, was very different from what it is for me. And looking at my son, first time I’m bringing 
him up today, so I need a medal. Anyway, if he has touched grass, for instance, of a certain 
thing, when he sees the grass, he doesn't just see lines of green. We obviously feel something 
with it. So along with what you're talking about, I look forward to being able to put visual 
information that can have rich meaning for us, but in entirely new ways or something, 


the two literal examples. That's all, and thank you very much for your answer. 


Brandel Zachernuk: Yeah. So you mentioned a neuroscientist. Was that Lisa Feldman 


Barrett®? Because if not then I’d love to know another one. Yes? Okay, good. Yeah, she's 
amazing in terms of her exposure to the way that priors are so important, in terms of what 
we're perceiving. So I’m glad we're on the same page there. 

Andreea Ion Cojocaru: Yes. She was recently on my Mindscape Podcast with Sean 
Carroll’, yeah. 

Brandel Zachernuk: So that, specifically, was on Mindscape? Okay, great. Thank you. And 
then, the next thing I wanted to talk about was, so I’m really glad to hear about your 
disinterest, potentially, and antipathy for immersiveness, for its own sake, because I share 
that. People who are regulars to this meeting know my hostility to the notion of story for its 
own sake as well. But you've also brought up being an octopus. So it strikes me that you 
would probably not consider being an octopus to be, sort of, significant in and of itself. But 
for some kind of functional practical benefit, some cognitive change that you would 


expect to occur. Have you played with Octopus? And what kinds of things have you 
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observed there? Are there any signs that you do different things there as a consequence? 


Andreea Ion Cojocaru: Yeah. So I use their methods. Giuseppe Riva is a researcher from 
Italy who is using VR and these theories of embodiment to treat our sort of mental 
conditions. And he has an onboarding protocol for helping people identify with an avatar. 
He's using it with hominoid avatars. But I’ve used that onboarding protocol, again, on myself, 
these are not things I make public or ever will, but on myself. You basically tap, you use the 
thing from the rubber hand illusion. You have someone tap your actual body, and then, you 
program something that will tap your other body in a place that's kind of in the same place. 
And then, I did an experiment to see the extent to which I can embody other kinds of stuff. So 
this tapping helps quite a lot to go into it fast. And I like to embody spaces. 

And this sounds nuts, but let's talk about it. I like to embody a room. | like to 
experiment with how big I can get. And again, this is completely crazy talk, but then here we 
are, in 2022, with VR in the hands of teenagers. So, yeah. It happens. I mean, it's real. How 
fast it happens and how profound that experience is will vary from person to person. It's kind 
of like, some people have lucid dreams, some people can trigger out-of-body experiences and 
some cannot. But the mechanism is there. And the technology now is there and costs 400 
bucks. Why do I do it? I’m interested in observing how I change. I’m interested in observing 
myself, and most particularly how I perceive physical reality afterward. So I’m trying to 
understand this transfer and see if I can have any kind of insight into that, then, I can phrase it 
in a more methodological way and start to form hypotheses. There are changes that are 
happening in me. I’m not at a point yet where I can talk about them with enough clarity to 
communicate them to other people, but they exist. 

And at the end of the day, I’m interested in what Foucault called, ‘Technologies of 
Self’. Because what I’m doing to myself is, ’m making myself the subject of technology of 
self, I’m using VR. But you can use other things that are not technically technology or not 
technology in the modern sense, you can use books or other kinds of things to push a change 
in myself that is very new. 

And I need to understand what I’m becoming. What's the possible direction of that? 
Because we might potentially face this happening on a global scale soon with very young 
people. And because scientists are so scared to talk publicly about this, they're so scared to 
throw things out there, because the VR developers are so scared to really go into this, we are 
left in a bad place right now, where we know we struggle. And I mean, I get a lot of shit for 
talking about these things. There's a lot of people telling me on Twitter that I’m wrong but I 
do think it's necessary, so I do it. 

I’m interested in how these things will change us, and what's the potential in that 


as well. I think it's even harmful to try to avoid it. So those developers working hard not to 
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trigger these things are harming everyone. The tech will do that anyway, so we might as well 
understand it and let it happen, or at least control how it happens. But we can't if we don't 
look at the mechanism. And I think that when these developers are talking about what they do 
to avoid it, they are not talking about the mechanism. They're not even trying. They're not 
hypothesising any mechanism that triggers them. They're kind of like band-aids, right? 
They're kind of like seeing something happening there and then they think it's something and 
trying to have local solutions for that. I don't know, did that answer your question? 

Brandel Zachernuk: Yes, absolutely. And your point about being a building I think is really 
thrilling. Reminds me of some stuff that Terry Pratchett, in Discworld, was a remarkably 
neuroplastic kind of writer. But it also reminded me of, when we were talking about the 
channels of information that we're using to, sort of, explore and mess with, that 
proprioception is completely distinct from visual. And to that end, the most exciting thing 
for me is virtual reality's capacity to impact what it is that we mean to do with our bodies, and 
what kind of impact that has. So it's very exciting to hear all of these things put together. 
Thank you. 


Peter Wasilko: I was wondering if you'd ever read Michael Benedict's 1991 book, 
Cyberspace: First Steps (Michael, 1991)? 

Andreea Ion Cojocaru: I did not no. Should I? 

Peter Wasilko: Yes, you should. It has very interesting presentations of abstract 
information spaces. And one of the ideas was, to have higher dimensional space represented 
as multiple three-dimensional spaces that can unfold to reveal nested subspaces inside. Sort 
of like, you're looking at three walls of the cube, then another sub-cube could open based 
upon a point that was selected within the first cube representing another three dimensions of 
the abstract information object. Also it introduced the idea that you could be representing a 
physical object in a space, but the space itself could represent a query into higher dimensional 
space. So the point in the space would represent the query corresponding to the three 
dimensions that were currently displayed in the one space, and that would then, control what 
was being displayed in another link space. So just the most fascinating thing I’ve read in a 
long time. And I keep coming back to that book and encouraging everyone in our group to 
take a look at it. So I highly recommend it. And when you do get a chance to read, I’d be 
extremely interested in what your reaction is to those chapters. 

Andreea Ion Cojocaru: I want to add something quickly. So the thing that crosses my mind, 
which again, it's not something I just say in public, but like, why not? Because today's 
discussion is already going interesting places. What crossed my mind, as you describe the 


book which I will absolutely read, is this: so let's say, I just said that I, sometimes, like to 
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embody an entire room. We can't understand these complex spaces and nested spaces on four- 
dimensional spaces and so on. But can we, if we are a room? What kind of perceptual 
possibilities and cognitive possibilities would that open up? Because, of course, if you 
truly believe that you are the room, your brain is in an altered state of consciousness, 
basically. Not in the like spiritual sense in any way, but at the cognitive of the cognitive level. 
So again, this is kind of wild speculation. But that's just the thought that crossed my mind. 
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Andy Campbell 


Dreaming Methods - Creating Immersive Literary Experiences 


Dreaming Methods has “always been at pains not to place text in front of images, or beneath 
them or to one side, like labels on tanks at the zoo or explanatory plaques next to pictures in a 
gallery... we explore to read. This avoids the danger of us regarding the texts as more 
important than the imagery. It pulls us in, and it makes [the] work inherently immersive and 
interactive.” — Furtherfield 


Campbell, 2022. 


How can text — when it changes from ‘static’ to ‘liquid’ in digital environments — become as 
absorbing and comprehensible to readers as traditional text? And what sort of effect can it 
have? 

Since 1999 Dreaming Methods has developed challenging and compelling works of 
digital fiction that blend text with immersive sound/visuals and explorative gameplay. These 
works often include experimental narratives-in-motion (animated, fragmentary, and multi- 
layered texts) which require different methods of both writing and reading. 

This short talk explains how our approach has evolved whilst maintaining a clear 


artistic vision: from early browser-based technologies such as Flash to ambitious narrative 
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games and VR experiences. We offer some fascinating insights through several real-world 
examples from our portfolio, including a virtual reality mobile library van/space shuttle 
designed to encourage children’s literacy and a spoken-word VR poetry experience currently 
shortlisted for the London Film Festival XR Prize that tells the stories of three Northern 


women. 


Video of presentation: https://vimeo.com/onetoonedevelopment/review/ 
753519382/02550aa9bf 


Presentation (pre-recorded for the Symposium) 


Dreaming Methods is a creative studio that develops immersive stories with a particular focus 
on writing and literature. We’ve been producing digital fiction for over 25 years. 

Much of Dreaming Methods’ early work was dark in tone and highly experimental. A 
mix of surreal dreams and urban horror, it was published online, mainly through Adobe Flash 
to shift away from the then quite tight constraints of HTML. My approach was to treat text as 
a visual and fluid entity, to challenge the reader to the extreme, to make the structure of the 
stories themselves something unreliable, unstable. 

We use a lot of the techniques that we originally developed with Flash to inform our current 
approach to digital fiction — especially when working in VR. 

WALLPAPER for example, part of a research project with Professor Alice Bell from 
Sheffield Hallam University called Reading Digital Fiction, is multi-layered in its approach 
to text. It’s an atmospheric and tense narrative with some surprising twists. 

The text within WALLPAPER appears on physical items within the gameworld, such as 
on postcards and letters to give a sense of grounding and normality, but it also has a ghostly 
presence: hand-written, circular, and floating like the cobwebs of memories; and as a flowing 
underlying texture that exists just beneath the environment’s surface. 

In The Water Cave, an explorable VR poem about depression, a single thread of 
glowing text acts as an umbilical cord through the entire experience, guiding the reader/ 
player out of the depths of the cave towards the surface, even though at times, ‘clinging to the 
words’ means having to submerge beneath the water. 

Digital Fiction Curios, which we created as part of another research project with Professor 
Alice Bell, is a prototype digital archive for VR that uniquely houses a selection of our old 
poems and stories created in Flash — a response to Flash being made redundant in 2020. 


Visualised in the style of a magical curiosity shop, readers/players can root around in 
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the environment, opening cabinets, digging into boxes, examining, and reading digital fiction 
from as far back as 1999. One of the most fascinating elements to this project is the ability to 
view old work in a completely new way. Curios also offers some re-imaginings of what these 


poems and stories might look like had they been created using today’s technologies. 


Our most recent VR work, Monoliths — a collaboration with Pilot Theatre, funded by 
XR Stories — immerses participants in the evocative tales of three Northern women through a 
series of surreal and atmospheric virtual spaces. This project treads a fine line between giving 
the participant enough imaginative room to visualise the stories, which are told through 
spoken word poems, whilst also making them feel as if they are existing within them. 
Interactivity is gentle and stripped back; during the final sequence, standing on a rocky beach 
at sunset, you’re ‘handed’ small, beautiful stones to examine as the poem flows. 

A common thread throughout all our work is a sense of immersion — we look to create 
portholes into self-contained, often short-lived worlds; dream-like environments where text 
manifests and stories are told in all kinds of intriguing and unexpected ways. It’s taken a long 
time for us to develop our voice and approach — and of course, it’s still evolving. Methods of 
writing are changing but so are methods of reading. That’s what we’re seeing right now, 


through our current projects. 


Links 


https://dreamingmethods.com 


https://dreamingmethods.com/portfolio/monoliths 
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Annie Murphy Paul 


Operationalizing the Extended Mind 


In the more than twenty years since the publication of the seminal paper by Andy Clark and 
David Chalmers titled The Extended Mind [2], the idea it introduced has become an essential 
umbrella concept under which a variety of scientific sub-fields have gathered. Embodied 
cognition, situated cognition, distributed cognition: each of these takes up a particular aspect 
of the extended mind, investigating how our thinking is extended by our bodies, by the spaces 
in which we learn and work, and by our interactions with other people. Such research has not 
only produced new insights into the nature of human cognition; it has also generated a corpus 
of evidence-based methods for extending the mind. My own book—also titled The Extended 
Mind [3]—set out to operationalize Clark and Chalmers's idea. In this talk, I will discuss the 


project of turning a philosophical sally into something practically useful. 


https://anniemurphypaul.com/books/the-extended-mind/ 
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Apurva Chitnis 


Journal : Public Zettlekastens 
The future of knowledge management on the internet 


These last few weeks I've been building my own Zettlekdsten®. It’s an intimidating German 
word, but the idea is simple: when you’re learning something, take many small notes and link 
these notes to one another to create a web of connected notes. This is more effective than 
taking notes in a long, linear form (as you might do in Apple Notes or Evernote) because you 


can see the relations between ideas, which helps with your understanding and retention. 


Zettelkasten. Clear, 2019. 


The core idea behind Zettlekastens is that knowledge is interrelated — it builds off one 
another, so your notes — your understanding of knowledge — should be too. Wikipedia is 


structured in a similar way, using links between related pages, and in fact even your brain 


fe 


stores knowledge in a hierarchical manner’. 


Limitations today 


But as powerful as they are, Zettlekastens implemented today are limited in two ways: firstly, 


they are only used for knowledge-work!®, and secondly, they only represent knowledge in 
your mind, and no one else's. These limitations are debilitating to the potential of 


Zettlekastens, and more broadly how we communicate online. 


I believe that not only knowledge, but all sentiment and expression is interrelated. 
Further, my knowledge and sentiment is built off of other people’s knowledge and sentiment, 


ie it extends beyond myself. 


For example: 


¢ [think that “NFTs are the future” after listening understanding “@naval’s belief that NFTs 


are necessary technology for the metaverse” in “this podcast” 


¢ Ilove “A Case of You” by “James Blake”, and “this is my favourite live performance” 
Public Zettlekasten 


So what would happen if we removed these constraints? Imagine if we each built our own, 
individual Zettlekasten, representing our thoughts, opinions and experiences, made them 
public, and related our knowledge and sentiment to each other. What could we do with that? 
A few ideas: 

* We could look back in time and see how someone we admire learnt about a topic. In the 
first case above, we can understand why @naval believes what he does about NFTs and 
the metaverse. We can see what influenced him in the past and read those same sources. 
Further, we could then build on his ideas, and add our own ideas, for example “someone 
needs to build a platform for trading NFTs in the metaverse”. Others could build off of our 


ideas, and others could follow their journey as they learn about something new. 


¢ We can understand how an artist we admire created something. In the second case above, 
we can see when James Blake first listened to the original “A Case of You” by Joni 
Mitchell, what he thought and felt about it, and why he decided to perform a cover. We 
could use that understanding to explore Joni Mitchell’s back catalog, or be inspired to 


create our own content, for example by performing a cover. Followers of Joni Mitchell 
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and James Blake could easily see our covers by following edges along the graph. 


These are just a few ideas, but if we each made our Zettlekasten public and interrelated to one 
another, then there would be as many interaction patters as there are people in the world. This 
would unlock new forms of consumption and creation that are not possible today. 

This knowledge and sentiment graph could be queried and accessed in a huge number 
of ways to answer a broad range of questions. You could effectively upload your brain to the 
internet, search through it (and those of others), and build on top of everyone’s ideas and 
experience. This is a new way of representing knowledge and expression that goes beyond 
the limitations of paper and Web 2.0: it allows us to work collaboratively, in ways that 


Twitter, Facebook and friends just aren’t able to offer today. 
Implementation 


What data-layer should be used for storing this data? A blockchain is one idea: the data would 
be open and accessible by anyone, effectively democratising all knowledge and sentiment. It 
would be free of any centralised authority - you could port your knowledge in whatever 
application you wanted to use, and developers could build whatever UIs make most sense for 
the task at hand. Finally, developers could create bots that support humans in linking and 
connecting relevant ideas to one another — a boon for usability efficiency and 


discoverability. 
Challenges 


The biggest challenge with this idea, if we use the blockchain as the data-layer, is that the 
information a user would create is public and permanent. You may not want the world to 
know you believed something in the past (eg if you were a fan of X in your youth), but you 
cannot easily delete data on the blockchain!!. You could, however, add a new note to explain 
that you no longer believe some idea — this would be particularly useful to any followers of 
yours, who now have additional context about why your opinion changed. 

Similarly, you'd be revealing all of a piece of knowledge or none of it; with a 
rudimentary implementation, you couldn't partially reveal a belief to just those you trust. 
Zero Knowledge Proofs might be a fruitful solution here. 

The second big challenge is how to present this data visually to end-users. Solving this 


particular challenge is outside the scope of this article, but it suffices to say that linear feeds 


a 


(such as Twitter or Facebook) wouldn’t work well. If these barriers could be overcome, 
public Zettlekastens could not only be how we represent knowledge online, but also how we 


understand ourselves and each other in the future. 
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Barbara Tversky 


Journal Guest Presentation : Mind in Motion 


https://youtu.be/RydjMrG9sDg?t=714 


So, thank you for inviting me. I have far too much to tell you. And I’m trying to tell it 
through visuals not in the book. The talk will be like pieces of hors d'oeuvres, so a bit 
disjointed, but they're meant to set up talking points so that you can ask questions, or 
discuss things. I should say that you're more punctual than my students, but my students are 
far more geographically dispersed. Kazakhstan, China, Korea, Japan, just everywhere. And 
so, "Zoom" does enable that kind of interaction in one class. 

I’m going to share a screen and I want to, before I show pictures, I just want to say a 
bit, without a picture, of how I got into this field at all. 

I’m a bit of a contrarian. When I was a graduate student, people were reducing 
everything that people thought about all representations of the world to something like, 
language, or propositions. And my feeling, looking at that, and I did look at all the research at 
the time, is language is efficient, decomposable, it has all kinds of advantages. I rather like it, 
I’m using it right now. But it seemed to me that language couldn't begin to describe faces, 
scenes, emotions, all kinds of subtleties. And then, I started thinking that space is half the 
cortex. So, spatial thinking must be important. And by spatial thinking I mean the world 
around us, and the things in it, including our own bodies, other people objects, scenes. And 
that special thinking evolved long before language, which occupies a rather tiny bit, but 
important place in the cortex, but came much later, and is in less connected with the rest of 
the cortex. And then you think, anyone who's been a parent, or owns a dog, that babies, and 
other creatures think and invent so many marvellous things without language. And for that 
matter, so do we. 

So, I got interested in spatial thinking. This is some of the early ways that we 
communicate. Gesture arises in children long before language. And in fact, children who 
gesture quite a bit, speak earlier. Games where we're imitating each other, taking turns, 
alternating what we're doing, this kind of interaction in games, rolling the ball, rolling it back, 
it builds trust. It sets up conversation, which is, you say something, I say something. So, it 
sets up cooperation, conversation, and many other things. This is done early on and 


communicated by action, not by actions of the body. And reciprocal expressions on the face it 
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isn't communicated by language. So, I’m going to jump lots of jumps, and I want to talk now, 


because you're interested in text, about kinds of discourse. 


¢ One important kind of discourse is a description, which is a state of affairs in space 
or time. 

¢ Explanations build on descriptions, but add a layer of causality and reasons. 

¢ Stories add on descriptions and explanations, they include both, and they add a point 
of view. The author, or authors, they add emotion, drama, there are a whole set of 
elements that stories can and each one isn't necessary. And there are deep arguments about 
what a story is. But I wanted to make those distinctions, and check with my colleagues in 
discourse analysis that the distinctions reflect the field. Stories are studied in so many 
disciplines. 

¢ Then we have directions, which build on explanations, but give you just an "A" to a 
"BM 

¢ And we have arguments, which bring together a set of stories, explanations, and 


descriptions to make a certain point. So, I’m not going to labour that. 


I want to jump again, I already talked about how communication begins in humans and other 
animals as well. Through the body, through the face, through actions. And I could talk, at this 


point, about mirror neurons, but Ill skip that, just leave it as a teaser. 


The earliest human communication and probably human includes Neanderthals and 
other hominids goes back at least 40.000 years. It keeps going back, as this was a 

discovery in the last few years. You can see hands there, there are animals there. It's from 
Sulawesi. And, as I say, these are being discovered everywhere. Sulewasi, 40,000 current 


oldest cave art: 
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This is the former oldest map, 6000 years. It shows two perspectives, an overview of the 
paths and rivers, and a frontal view of landmarks. Linguists don't like this. Geographers 


don't like two perspectives. But people seem fine with them. Ancient Babylonian clap map: 


This is the current oldest map, it's about two inches by one inch. A stone. It shows the 
surroundings around the cave where it was found, some 13.000, 14.000 years ago. And it's 
tiny. So, it could be taken with you, to guide you on going back. Map on stone block, 
southern Spain, 13,660 years ago: 
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A map of the sky going back 5000 years. Sky Map of Ancient Nineveh 3300 BCE: 


This is a valley in Italy, it's a drawing of a petroglyph. Again, two points of view. Bedolina, 
Italy, 2000 BCE: 
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Eskimo maps. They were carved in wood, very beautiful, carried on canoes, they showed the 
outlines of the coasts. And they floated, in case they fell in the water. Eskimo Coastal Map: 


South Sea Islanders Map, probably familiar to you. Shells representing islands, bamboo 
strips, the ocean currents, which are like the highways of the ocean. And at least some of the 
people that were trained and carried these with them, 2.000 miles on the open ocean, at least 


some of them returned home. South Sea Islanders Map: 
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A map by North Coast Indians, showing the various settlements on their hands. A map by 
North Coast Indians: 


Now I’m jumping again to depictions of scenes. Again, going back 40.000 years. Chauvet. 


Going back even farther in Sulawesi, although I’m not sure. Chauvet Cave 40,000 years ago: 
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This one I especially like, it is in the book. It's a petroglyph on the left, and the drawing of it 
on the right. And it's showing two suns in the sky. Quite remarkable what could account for 
that. An Indian astronomer did some history on it and found that, at about the time they could 
date the petroglyph, there was a supernova. And it was such a remarkable event that someone 
inscribed it in a stone. Stones were, in a way, the newspapers of antiquity. Supernova: 4000 
BCE Kashmir: 


Here's another example from the U.S., a whole valley full of these. It's called Newspaper 


Valley, and it has many of these petroglyphs showing events. 'Newspaper' Valley: 
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So, events in making bread in a tomb in Egypt. Bread making in Egyptian toomb: 


Events in the Trajan Column. Trajan Column: 
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Now we have calendars, they also go way back. Some circular. Some tabular. Calendars: 


Mayan 5" BCE? Egyptian 3K BCE 


All these forms become important, but I won't be able to talk about them. Then we have 
number. We have tallies. Again, you can find them all over the world. It's not clear what 


they're representing. Incised ochre tallies Blombos Cave, South Africa 70-100k: 
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But having a one-to-one correspondence from a mark, to an idea, to an object, to people 
whatever they were counting, moons, is a rudimentary form of arithmetic that was again, 


inscribed in stone. 


So, ancient visualizations represent: 


¢ Space 
¢ They represent people, objects 
¢ They represent time, and events in time 


¢ They represent number 


These are all important concepts, and you will find them in the newspapers, journals, 
magazines of today. And they're so important that the brain has specialized areas for 
processing them. And what's extraordinary about all of these is, they can be spatialized. So, 
this is part of my argument that is, spatial thinking is foundational to all thought. 


Early communications began as pictographs. In some way, you can still find... Well, there 
was a civil war colonel who collected these during battles and then, Dover later printed his 
findings. They're quite remarkable. This is a love letter between the two animals. On the left 
are her totem and her lover's totem. And it's a map leading him to her tepee, and she's 


beckoning him there in the map. Love letter: 
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FIG. L1.—-LETTER FROM AN OJIBWA GIRL TO HER LOVER 
From Mallery, Picture- Writing, p. 363 


In the 18th century, the age of enlightenment, we finally get graphs. 


Exports and Imports to and from DENMARK & NORWAY from 1700 10 1780, 


1700 1710 1720 1750 1740 6 177° 


The Bottom line is divided into Years, the Right hand line into L1Q000 each. 
Published a0 the Act directs, 1 May 1786. by WO Playfair Neale sculpt IDL, Strand, Lender . 


Trade-Balance Time-Series Chart. Playfair, 1786. 


Because the early visualizations, that ones that I showed you, except for time, were more or 
less things that were actually in the visual spatial world. But more abstract concepts, like 
balance of payment and graphs, developed only in the late 18th century, and they began to 
blossom. So, Diderot, I would love to walk you through this, it's a way of teaching diagrams. 
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The top half is a scene, which would be familiar to 18th century eyes. The bottom is a 
diagram. It differs from the top, and things are arranged in rows and columns. There's a key. 
Lighting is used not naturally, but to reflect the features of the objects. The objects are sized. 


So, you can see them in the diagram, not in their natural sizes. 


Pinmaker '’s Factory. Diderot, 1751. 


So, this is a visual way of teaching people what a diagram is. In fact, by now, we've 
diagrammed the world, and we've set up where different kinds of vehicles, pedestrians can 
go, where they can't go, where they can park, when they can go, and it moves us through 


space in an organized way. But we've really diagrammed the world. 
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Diagrammed the world. Tversky, 2022. 


Graphics augment cognition, they: 


* record information 
* convey information 
* promote inferences, enable new ideas. 


¢ This is a lot through sketches, and I won't be able to talk about that. But can answer. They 


facilitate collaboration. 


They're public, so we can both, or all of us, revise them, make inferences from them, enable 


new ideas from them, see them, and point to them. Gestures are important. 


We can distinguish two kinds of graphics: 


¢ Those that are inherently visual. Maps would be a prime example, they're ancient 


¢ And visualizations of metaphorically visual. Graphs, charts, diagrams 


And again, they seem to be a Western, at first production of the age of enlightenment. 


Good graphics schematize. This is a prime example. They also annotate. They're multi- 
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modal. 


Graphics consist of: 


¢ elements 


* relations among them 


And my argument is, these can convey meaning directly. But they don't have to be 
learned. In many cases, they're quite natural. One of my claims is this visual spatial way of 


communicating is much more natural. 


Pointing. Pointing draws a line from my finger, to the object that it's pointing at. So, it 
guides your eye to that object. And that's very natural. We saw babies use them. Chimps use 
them a bit, but other chimps don't follow them. It shows what the chimp is thinking, but it 


isn't taken as a communication by the other chimps, those can be separate. 


And then again, we've done a lot of work on gestures, showing that many of them are 
really helping us think. But they can also help others think. So, graphics consist of elements, 


spatial relations among them, they convey meaning quite directly, and they: 


represent thought directly, by using 


¢ elements and spatial relations on a page, a virtual page, or a page in the air, as gestures do 


* represent elements and relations in the world 


Elements can resemble what they represent: 


¢ Iconic. They can bear a conceptual similarity to what they represent. We call them figures 
of depiction 

* Metaphoric. And they can be schematically related 

¢ Abstract 

¢ Symbolic 


Early writing began as pictographic. And the alphabet was invented only once and took 
over the planet. Not everywhere, actually. Probably half the planet is still using some 


descendant of the pictographic language. 
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Meaningful marks. Our claim is that icons, dots, or depictions, lines, arrows, 


containers, can represent ideas quite directly, and we have some empirical evidence for it. 


icon thing 
point wo place, idea 
line mmm relation 


arrow >, @ «asymmetric relation 


container Oo [ ] set, area 


And Ill show you those in a minute with a detour to a minimal diagram, or could be called a 
minimal diagram, a line and a dot namely points representing ideas or places in a real map, 
and lines representing relations among them. And this minimal diagram, a link between two 


points, is the building block for many. 


This is the internet, in 1987, you could still draw it. Family trees. Social networks. These are 
social networks produced by some of our participants. Phylogenetic trees. Art. This is Mark 
Lombardi, no longer with us, who made beautiful networks representing where the money 


went, and other political, economic ideas. And people scrutinized them for hours. 
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NSF net 1987 


NSF Net Phyucal Connectivity -- April 87 


* For tome networks internal vructure (eg Wwonett) 4 wpprened ow we 


So, the meanings of these abstract forms are: 


simple, efficient, neutral, and abstract 


¢ Line: 1D, path, link 

* Cross: intersection of 2 paths 
¢ Arrow: Asymmetric path 

* Blobs: Enclosures, area 


* Circles: Cycles 


Some of the meanings come from Gestalt Principles, some just from gestures, some from the 
way we behave in the world. Like the paths on the ground are the lines that people make from 
place to place. So, we've done empirical work on each of these, showing that people produce 
them from verbal meanings, and they understand the graphic meaning from the verbal. And 


I’m not going to be able to go into that. 


94 


Spatial relations (graphics, gesture, speech) can be: 


¢ Literal 


* Metaphoric 


¢ Proximity in space represents proximity on any dimension 
¢ Place: center, periphery 
¢ Direction 

- Vertical: gravity, good, strong, health, wealth 


- Horizontal: cultural 


The general inference that people make is, proximity in space represents proximity on any 
dimension. We use this in gesture and language. We say we've grown closer to people. We've 
grown far apart. Place, centre, periphery. Again, language represents that the centre is the 
centre, and the periphery is the periphery. 

Directionality is also important. The vertical. Anything that goes up is good in general, 
except the economist thought inflation going up, and unemployment going up. I could say 
they're perverse, but it's probably because of the numbers. We often get conflicting concepts 
wanting to go up. But this is gravity. Fighting gravity takes strength, health, wealth. So, 
anything going up is usually positive, and anything going down, like hell, is negative. 

The horizontal is pretty neutral. It's neutral in our lives, in our motion, in our world. But 
there are cultural constraints on it. Writing order that are quite strong and cross-cultural. 
There are cultures and languages like Hebrew and Arabic written from right to left. Originally 
the columns in Japanese and Chinese went that way. And many of these concepts get reversed 
in right-to-left languages. So, just to show you, we looked at diagrams in books, all in many 
disciplines. And the present day, or better things go up. And I should say, in evolutionary 


trees it's always man. Who gave birth to man? 
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So, another thing we've looked at, and others of you have looked at going to graphs, now to 


information graphics is inferences. 


B 
Y Y 
A 
x x 


Different displays -> Different inferences. Tversky, 2022. 


¢ Bars: discrete comparisons 


¢ Lines: trends 


This research needs to be done more, that different displays lead to different kinds of 


inferences. And the reason really is the underlying visual spatial representation. 


¢ Bars separate. They say there are a bunch of "A's" in this container, and a bunch of "B's" 


in another container, and therefore, encourage discrete comparisons. 
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¢ Lines connect showing a relationship. They say "A" and "B" share a value, but "B" is 
higher than "A." 


And we've shown these effects in a number of different contexts. Despite what the 
statisticians would recommend, the visual form of the displays override it, and people tend to 
make discrete comparisons from bars and trends from lines. So, this kind of research is 
ongoing, of how different displays, depending on the visual spatial characteristics, lead 


people to make different inferences. 


Animation. People ask me about animation. Animation is something that's relatively easy to 


do now. 


Horse in Motion. Muybridge, 1877. 


There's tons of research showing that people don't understand animations the way they're 
intended. The Muybridge experiments looking at, whether all four legs were off the ground at 
any point when horses were galloping, as an instance. You can't see it when horses are 
galloping. But the stop gallop photography showed that, yes, all four legs are, at some point, 
off the ground. But the art museums of the world are filled with horses legs incorrectly 


aligned when they're galloping. Here's an example of how hard it is to perceive. 


Conveying change over time: 


¢ Animations use change in time to convey change in time 
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¢ But: 
- Animations are hard to perceive 
- Animations show but do not explain 


- Animations conceived as discrete steps 


Animations are compatible with thought, in the sense that, they use change in time to convey 
change in time, but they're hard to perceive. They show but don't explain. And most of the 
things that are animated, when we talk about them, chemical processes running, climate 
change, we talk about it in steps. So we think about these things in discrete steps, not in this 
continuous way. Which, as I’ve tried to show you, is difficult anyway. I’m sure good 
animations can be designed, but it's trickier than some people think. And obviously, 


animations appeal to the eye. We're all, in one way or another, addicted to movies and music. 


Comics. I want to jump to comics because they show all kinds of lovely ways of expressing 
meaning that are rarely seen in traditional graphics. 

Whether they're infographics or graphs and charts. So, one thing comics artists can do, 
is use space to segment and connect time and space. Here you get an overview of the scene, 
and then you get the action superimposed on it, in frames on it. This was used by the ancient 


Aztecs, not just modern comic artist. 
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Gasoline Alley. King, 1918. 


Visual anaphora. You can get from one frame to another following this red book. The "New 


Yorker" cover is not just showing writing, but it's a visual story and a pun. 
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THE a 


NEW YORKE 


Shelf-life 


New Yorker Cover. New Yorker, 2008. 


Visual anaphora provides continuity over changes in space & time. So the book ends up 
in a trash can being burned by homeless people to keep warm, and the verbal name is "Shelf- 
life." But you can follow it because of the anaphora provided. Something from frame one is 
preserved in frame n-plus-one. And so on, so that you can follow the continuity. As for good 
stories and good movies, often you want to break the continuity to create suspense. 

Here, following the eyes, and the pointedness of the frames allows you to go back and 
forth and understand the David and Goliath. 
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This one's a little harder. It's a beautiful book called "Signal to Noise" by Neil Gaiman and 
illustrated fantastically by Dave McKean. 


Film faces. Actors. Directors 
Patras. Old faces Ive bought 
te film and junk shops oo 
three continents. A patchwork 
of the nametess and the ones 
that interested me, with, here 
and there, a sprinkling of 
tare. 


They are ary frame of reference, 
the workd in which I move. | can 
stare at them for hour, wondering 
about the people behind the faces, 
their lives before and after the 
frozen second they are trapped in. 


Signal to Noise. McKean, 1989. 


It's showing an aging director, and he's actually dying of cancer, and he's got photographs 


from many of his productions on the wall. You can see he's thinking. And it shifts perspective 
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to what he's thinking about. He's looking, and you can see the perspective switch between the 
man in the blue coloured shirt and what he's thinking about, as he watches, looks at all of 


these frames, and then finally, he can't stand it. "Stop looking at me!" 


Signal to Noise 2. McKean, 2022. 


So, he's both reviewing his life and haunted by it. And again, it's conveyed visually. 
Steinberg, the master, a conveying peeping toms through a mirror that reflects the guy 
watching from the opposite apartment. 


More Steinberg. A pun, "Time Flies." More comics (only one shown). 
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Don't. Steinberg, 2969. 


So here, there are so many devices, visual spatial feed metaphorical, or figures of depiction. 
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You have puns here, polysemy, figure/ground. I want to draw your attention to the old- 
fashioned telephone cord, which some of you, at least, will remember. So this woman is 
drawing those other people into a conspiracy by calling them on the phone. So, the phone 
cord is a literal phone cord, it's a metaphoric phone cord, drawing them into the same 
conspiracy. It also serves as the frames of the panels. So it's doing triple duty. It's something 
kids can get right away. Like gestures, you get it almost without thinking. It just pops out at 
you. So, a beautifully crafted device. 

Figure/ground. You're seeing the murder, and the noise of the murder is coming 
through in those black figures that are superimposed on the actual scene. And the black and 
white drawing is emphasizing the stark brutality of the punctual murder. You light out a life 


in a second. 
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(Not illustrated) More Steinberg. "Canceling Thoughts." Again, I don't need to tell you. 
Visual juxtaposition. This is another Gaiman, McKean cooperation. A child is at a birthday 
party. You can see on the right, they're playing musical chairs. Here, the child is not interested 
in the birthday party. So, goes out, and talks with an uncle, who told the child the story of the 
Saint Valentine's Massacre by Capone, where he tied his enemies up on chairs and killed 
them. Shot them one by one. So, you have the chairs there with the men chained to the chairs 
juxtaposed with it with the birthday, which is a little bit of a brutal game because one child is 
eliminated at each round from musical chairs. So, that juxtaposition of chairs, again, is a stark 
reminder of the comparison between brutality of children, and brutality of adults. 

(Not illustrated) Okay, metaphor pun. "Puppet Governments," Feininger. This is Winsor 
McCay, a brilliant comics artist. This is from the early 1900s, New York. Parts of New York 


still look like that. And this is, of course, the rat race, running on a treadmill. 


(Not illustrated) This is a dream, another one of his where a dream transports the child and 
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then dumps the child back in bed, the way dreams end before they should end. 

(Not illustrated) This is onomatopoeia rhythmicity. It's showing a chase. And by putting the 
panels on a diagonal, showing the speed of the chase. "Coming out of the frame." The first 
pig, whose house was blown up, comes out of the frame, and talks to this second pig inside 
the frame, and says, "Get out of there. It's safe out here." And then, the pigs all go berserk. 
They get out of the frames. And the frames are on the floor, and they're stamping on them. 
So, this version of the three little pigs is a riot. And again, kids can get it. 

I’m going to end with another Steinberg. Steinberg drawing himself. Again, a visual way of 
understanding drawing portraits and so forth. So, I’ve raced through a lot, and I haven't 


covered everything that Frode wanted me to talk about. 
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Brandel Zachernuk: Hi, Barbara. Thank you. This is brilliant. I love the fresh ground and 
the emphasis on the text here. So, my question is not about animations per see, but about 
progressively recomposed images accompanied by illustration, via the speech of the 
illustrator. The actual drawing of lines along with narration. Is that something that you've 
ever studied, or that you would expect to have any particular effect from, in contrast to seeing 
the completed image of its entirety? 

Barbara Tversky: That's what we do in classrooms, right? I mean, that was the oldstown, I 
know I have many mathematician friends who still insist on going on the board as they speak. 
And watching it unfold, and the rhythm with which it unfolds, and the verbal accompaniment 
at the same time, I think is very effective. So, what you're pointing to is one way that 
animations can be made more effective. They unfold in time with narration and explanation. 
And they add a bit of drama. What's going to come next? So, I think that's great. And at the 
back of my head, when I was thinking about this is: What can you do on text? And it will 
amplify it. And that is exactly the sort of thing that one can do. And it is like a comic, 
combining language, and symbols, and sketches, and so forth, all at once. There are beautiful 
examples on the web. Just lovely examples of people using that technique. And I’ve been 
teaching comics for probably 20 years, on and off, not quite. And I see a younger generation, 
growing up with that medium, and drawing and writing at the same time. So, I think people 
will get adept and talented at doing that, at illustrating what they think, while they're thinking 
it. And I think that's just great. It gives people an extra way of expressing themselves that's 
quite poetic, or can be quite poetic, but it's also wonderful explanations. So, yeah. I’m a real 
fan of that. 


Brandel Zachernuk: I’m curious, have you ever seen Ken Perlin's work at NYU around at 
being able to draw in Virtual Reality? 

Barbara Tversky: I was an early fan. And Ken, as a friend, and I was an early fan of his, 
exactly on chalk talk. And in fact he and I and Steven Feiner, whom I work with, and Hiroshi 
Ishii at MIT, the four of us put in grant, after grant, after grant to expand, and NSF didn't like 
it, and didn't like it, and didn't like it. So, a real disappointment for all of us. Because 
classroom teaching that way is, again, natural and what Ken's animations did is, you're 
talking about a pendulum, and then it could animate the pendulum depending on the length of 
the string, and so forth. So, being able to speak, and use mathematical mathematics driven 


animations, I thought was super! Just a super way of understanding. So, yes. 


Bob Horn: Oh, hi Barbara. Of course, the question I will ask will not be a surprise to you. 


I’m very interested in, and wonder the degree to which you've done research on the textual 
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elements intimately integrated with the spatial elements. That is most of what you've just 
presented has been the spatial aspects of the kind of visual language communication that we 
are all using. In addition, the diagrams and comics rely, it seems as much as maybe, 50/50 or 
even more sometimes, on the words, and how the words are integrated with the visual 
elements. And that's been something that I’ve been very interested in, particularly in 
diagramming. So, I’m wondering if you've gotten your research to go in that direction, to 
analyse, and find out how text is integrated with the visual elements? 

Barbara Tversky: We've done a lot of work that skirts around that. We've shown that you 
can go back and forth between visual descriptions of maps, or many kinds of diagrams, and 
the visual spatial. That the same underlying concepts are driving both of them. But that the 
visual form, for example, root maps is usually not for everybody. But usually a more effective 
way of communicating that. It's a long story. But I agree with you that in many comics, 
what's going on is in the words. I think they're poor comics when they're talking heads. I talk 
about them as talking heads. If you look at Larry Gonick's science and history, if you look at 
his comics, they're cheap, 10, 12 dollars each. They're absolutely wonderful. His book on 
statistics is used as a textbook in many places. At one time, even Stanford. And he's a 
neighbour in San Francisco, and his books are absolutely fabulous. He was all about 
dissertation mathematics at Harvard. A self-taught cartoonist. And he began doing, 
essentially, visual spatial textbooks on different forms of math, science, history, sex, 
environment. He's got bunches of them. He always works with a domain expert. We've 
appeared together on many occasions. And once I had the temerity, stupidity to ask him," 
Larry, what do you put in pictures? What do you decide to put in pictures? And what do you 
decide to put in words?" So, he's very tall, I’m not, and he kind of looked down at me, with 
his full height and said, "Barbara, I do everything in pictures. What I can't do in pictures, I do 
in words." And he's incredibly inventive of what he does in pictures. I have my students go 
through one of his books, they each choose one, and they go through looking for the visual 
spatial devices, and every year they come up with things that I haven't thought of. They see 
things I don't. And it's usually the visual spatial telling the story. So he's an excellent example 
of that. There are others. And Scott McCloud and his book "Understanding Comics" is a gem. 
It's a gem about stories and narratives, not just about visual stories. But he does talk about the 
roles of language. And here you'd have to add symbols, like arrows and mathematical 
symbols. You have to add in comics the way the font, the size of the font, and all the 
squiggles that are added that give you information about movement, and mood, and smell, 
and sound. So, you can enrich the depictions in so many inventive ways. And what I’ve been 
trying to do is urge people who make charts, and graphs, and infographs for science books, to 


learn those techniques. And, as I say, I’m just pleased to watch younger people. I have a 
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sample of eight grandchildren, and the grandchildren of many of my friends, and watch them 
latch onto graphic books, and see the graphic books that are doing so much in the depictions 
besides text boxes, and I’m very optimistic about people coming up with really creative ways 


to do visual storytelling. So, long answer. Sorry. 


Ben Shneiderman: Thank you, Barbara for a wonderful, intense, movement through all the 
space of these wonderful ideas. I think it aligned very well with Bob Horn's visual language 
thinking, which has been an inspiration for me as well. But one of the charms of your book 
was that, it went beyond the spatial and the visual, to the idea of mind in motion. And could 
you say more about dance and body movement? You talked about gesture, you talk about 
hands and how people communicate, express themselves, learn by being in motion. Tell us 
more about that side of the story. 

Barbara Tversky: So, it means speech is in motion. And speech is accompanied by prosody. 
I emphasize certain words, and de-emphasize others. I can give you my mood by, I can sigh, I 
can sigh short or long, and that's motion, and it's just in our voices, and it's communicating so 
much more than just the words on page. Although, text again is words on page. And there are 
ways of amplifying text by putting "dot, dot, dot, dot," that capture some of that. And sure, 
our bodies indicate, I mean, I said gravity, if I’m feeling good, I’m standing straight and 
strong, and if I’m in a depressed mood, I’m down, and we can pick that up in others in a 
second, especially people we know. We can pick it up from hearing their feet behind us. What 
kind of a mood they're in. Who it is. We know these cues. Again, they're active, motion cues 
to people. They're very simple, not as complicated as dance. But really creative, and 
wonderful dancers, and choreographers can create absolutely amazing displays of emotion, 
human interaction, human non-interaction, individual feelings from the way they do dance. 
They're uncanny. And you see that in theatre, they often hire choreographers to orchestrate 
how people are moving, and talking, moving their arm, agitated or smooth. So, yes. Huge 
amounts of human meaning gets conveyed through the motion of the body. 

Ben Shneiderman: I do think really that deserves on a much-expanded part, just the idea of 
walking together, being in a forest, moving forward, sailing on an ocean, flying through the 
air, walking up a mountain. All of those to me, they're not just physical experiences, they're 
cognitive experiences as well. And they enrich us. And I found your book really opened my 
mind and thinking to the realization of, how much the body plays a role. Which your book 
adds so much to enrich the dimensions of analysis, which have been, as you point out, largely 
linguistic moving towards spatial, and visual, and maybe auditory. But the idea of moving 
towards body motion that was really, to me, a highlight. 


Barbara Tversky: Thank you. I was limited in the book by what there was research on. This 
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is the problem of being a scientist. You don't want to go too much beyond research. I use a lot 
of examples, but the examples are all founded in research findings. But I couldn't go off the 
way until now. But really, if you think about it, every organism, even a virus, needs to move 
in space to survive. And the basic movement is approach or avoid. And those are replete with 
emotion. You approach things that you're attracted to, that might do you good, that you want. 
You avoid things that have negative valence. So that, from the get-go, movement is for 
survival. Even grasses have to move toward the sun or away from rain in order to survive. 
Even things rooted in the ground. So, we all have to move in space to survive. The basic 
movement is approach or avoid. And those come with emotions, which I think underlies 
some of Damasio's claims, although he's got brain there too, without emotion nothing 
happens. And emotion and motion in English and other romance languages have the same 
root. I don't know about Germanic, or Chinese, or other languages. But they do have the same 
root. And we talk about being moved as an emotive response. So, I do think anything that has 


to do with life, really does derive from motion in space. 


Ben Shneiderman: Exactly. But look at how they impact on design, or even the "Zoom" in 
front of us. Some people have just their text name. Some people have a frozen image. Others 
are live and animated. I like to be Zoomed standing up, so I can be freer to move around. And 
I think I express myself better, and I can reach out, or I feel the other person better when they 
are animated, as well. Anyway, thank you. 

Barbara Tversky: Yeah, absolutely. I’m frozen in place in a classroom, I can't stand. But 
when I move around the classroom, I’m going the whole width, and sometimes the length of 
the classroom. So, I can't do that on "Zoom," it drives people crazy. So, I plant myself in a 
chair. And when I’m listening to "Zoom," it's often on my phone, walking. So, a longer story. 


"Zoom" has advantages and disadvantages. Like anything. 


Frode Hegland: Yeah, absolutely. You mentioned the question of other languages like 
Germanic languages. In Norwegian, "folese" is the word for feeling. But that can be, you can 
touch, it's also touch feeling, as well as an emotional feeling. But the funny thing is that, 
"bevege seg," which means movement, is also what you would say if you were emotionally 
moved by something. 

Barbara Tversky: Yeah, nice. I should ask my students who are in Japan, or China, or 


Kazakhstan, or Malaysia what their languages do. Yeah, thanks. 


Peter Wasilko: Yes. Have you given any thoughts to the evolution and interplay of note- 
taking? And I was recently reading "Lines of Thought," (INDISTINCT) typesetting and 


textbook design. And also, do you have any thoughts about interactive fiction systems? 
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Barbara Tversky: I’m having trouble understanding you. Did someone understand him 


better and can tell me what the question is? The mic is bad, my hearing is shut. 


Frode Hegland: Well, I got the last part really clearly, Peter. You asked if Barbara has any 
comments or perspectives on interactive fiction? Can you please repeat the first part? I also 
had some problems hearing you. 

Peter Wasilko: Yes, the first part was whether she had any thoughts about the evolution of 
note-taking and textbook design? 

Frode Hegland: The evolution of note-taking and textbook design, as well as how they 
interplay. Thanks, Peter. 

Barbara Tversky: | don't really know much about the history of textbooks. I do know there 
are a huge number of experiments trying to compare text and diagrams or graphics in one 
way. And many of them are unsatisfactory, because you can have good text, or poor text, and 
you can have good graphics, or poor graphics. And I think people in the info design graphics 
community have been developing standards, or best practices. For good graphics is 
complicated, because it depends on your audience, and what you're trying to convey. You 
can't have absolute principles like you can for font size. And there's a little bit of work on 
textbook design and what it should look like in good text and poor text. But the range, in both 
cases, seems so great. And studying it would take an historian of sorts, to know the evolution, 
the development of those things. And it would have to go across cultures. What happened in 
the East, as well as what's happened here. So, I think that's way beyond my expertise. but I 
don't think I answered all the parts of the question. Probably I can't answer them. 

Frode Hegland: The second part was on interactive fiction. 

Barbara Tversky: Ah, interactive fiction. I don't know whether people have done research 
on it. I know my kids, who are now parents themselves, loved it as a kid. They weren't 
around when I was a kid. But my kids loved it. And then, of course, the computer games that 
are built on storytelling. People get incredibly involved in. So, probably those designers 
know a great deal. They have a lot of heuristics and rules of thumb for that. So, the one kind 
of discourse I didn't put up is conversation. And that grounds interactivity. And conversation 
isn't like a lecture, it's two or more people speaking and no one can dominate. As I’m 
dominating now, in a normal conversation no one can dominate. What you get in that kind of 
interactivity also, is little bits of information. Bite size. That you can consume, and it arouses 
a question, and then there's another bit of information. And interactive graphics do that. They 
allow you to involve yourself in it, in little bits, where you can get background where you 
need it, or where you want it. Not all of us want, you know, there's that old joke about a book 
about penguins that told me more about penguins than I ever wanted to know. So, different 


people will want different amounts of information. And that interactivity allows me to have a 
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conversation with a graphic, where I’m asking bite-sized questions, and getting bite-sized 
answers that lead to something else. So, I’m building up my own knowledge that way. And I 
think the interactive fiction can do that. And also add the suspense to it. We started at one 
point trying to compare comics with traditional graphics, or traditional graphics plus text. Too 
many things were going on at the same time. Too many uncontrolled variables. And as a 
cognitive scientist, it's those crucial variables that we're after. When you're a designer, or an 
educator, it doesn't matter to you what's doing it. The combination is probably doing it. It's in 
the interaction, amongst those elements. So again, then finding guidelines for creating good 
ones becomes difficult, because there are so many moving parts. I mean, like building a city. 
But nevertheless, we can judge which ones are more and less effective and why. There are 
times when I want to lecture or a book. There are times when I want that interactivity. Again, 


I’m not sure I’m getting at your question, but. 


Alan Laidlaw: Sure. I’ve got so many questions. Thank you so much for giving this talk. 
And I had the childish desire, which I still may succumb to, of showing off every reference 
you made. That's somewhere behind me because I’m a nerd. But that's great, I used to be a 
cartoonist, it's where I got started and a lot of that came from reading "Understanding 
Comics" and that was my first entry to like, "Oh, this person thinks like me." And I’ve never 
had these thoughts. But enough of that because could go on with many questions. So instead, 
Ill throw one that just popped up while you're giving the talk. This may be out there, and feel 
free to dismiss it. The thinking in context of cave paintings, and sort of where we got started 
in scribing. The commonality, seems to me, that it's always the physical act of resistance 
against a surface. And so, I’m wondering about that in context to where a theme has been 
trying to probe into VR, what that'll be like? Is there any research around how that resistance, 
that pushing against something to create is different than, or I don't know, is it a class? 
Because with VR, we could say at least, there's nothing to push against at the moment. But in 
dance, there's also nothing to... Well, there are motions, there's creation that doesn't involve 
resistance exactly, not in the same sense of pushing against something to. Does any of that 


make sense and is there any work on it? 


Barbara Tversky: Yeah, thank you. And I could probably learn a great deal from you, as a 
cartoonist. I absolutely agree with you. I don't know about research, but it's one of the 
complaints that people in architecture schools have, that people no longer know how to draw. 
That drawing on pixels is just very different from using a pencil, or a pen, or a brush. And 
artists, and calligraphers, and so forth talk about what the thing is, how it's held in their hand, 
what are the motions they need to do. Cooks. Any of you who cook knows you have certain 


knives that work well with your hands, and others that don't. Resistance and dance is gravity. 
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And your own body, what it can do. will it stretch enough or not? Does it have the strength? 
So, that feedback to the body is huge. 


Alan Laidlaw: I guess I put it... Sorry to reframe that, the aspect of us versus surface is what 
I was kind of trying to... The creation always seems to have a surface that's separate from us. 
Anyway, continue. 

Barbara Tversky: Yeah. I mean, I’m trying to generalize that to resistance, and feedback to 
the body, and the feel that it is when you're dealing with a surface. And again, different 
surfaces. Just writing, paper makes a difference. Which kind of paper you're writing on? Or 
doing charcoal on, or watercolours, all of that. And that, I think, it's more than the resistance, 
it's the subtlety of your hand movement, and wrist movement, and our movement on that 
surface, what it takes. And in calligraphy, they practice for years the strokes, and how they 
make them, and how they twist the brush, and the kind of paper. So, all that interaction with 
the medium, what it gives your hand. And artists, I worked with a bunch of artists interested 
in drawing, and some of them had done doctoral thesis, and one of them looked at 
professional artists, and accomplished artists, and novices on drawing, how much they're 
looking, and how much they're drawing, and what are the time spans of the interaction. And 
in artists, it's much longer. They can look and draw a lot. And look and draw a lot. Novices 
are going back and forth. So, for artists the knowledge is already in their hand of how to 
translate what they see, this is life drawing, into their hands. And they talk about it as a 
conversation between the eye, and the hand, and the mind. And if you try to get them to talk 
words at the same time, they can't do it. The words get in the way. It's a visual, spatial, motor 
conversation that the words get in the way. And architects say the same. They can talk 
afterward. Explaining what they were doing from a video, but while they're doing it, they're 
deeply engrossed in this feedback loop. Does that align with your experience? 

Alan Laidlaw: Yeah, to play off of that, that's actually great. And got me thinking that now 
we have keyboards as our main interface. Which is a sad state compared to the richness of the 
ideal, the nostalgia, for calligraphy and whatnot. And yet, we have translated our focus into 
the simple clicking of buttons at a repeated pace and moving a mouse around. We can still get 
to that flow state, right? Coders do it, etc. So, that gives me hope in the VR space that, even 
though we wouldn't have a surface to push against to create, we would still find a way to 
translate it through, I guess, just mainly the feedback, and the style of feedback travesty. The 
style of feedback would still come through, and we would still have that feeling. I was just 
wondering if there was something haptic, like in the way that we have gestures. I think 
Darwin said that, "Every culture does this." Some version of this to say, "I don't know." And 
if there was something about the creation of man that is pushing against something, and that 
equals the brain does something different then? 
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Barbara Tversky: Sure. I mean, the feedback, and the kind of feedback, and the mode of 
interactivity, and some of that, I mean, VR is trying to add the kind of haptics feedback. and 
you certainly need it for surgery. And the VR surgery does try to add haptics, because 
anything you do, as a surgeon, you're relying on that. And anything a cook is making. And it's 
how it feels, you need that feedback. And the interactivity that comes from touching and 
moving, you need it for taking care of babies. When you pick up a baby ant the baby is tense 
or relaxed, you feel it in your hands right away. So, yeah. We need that level of interactivity. 
Smell is another thing. When I cook I’m relying on the smells to know, I got three or four 
pots going and I’m relying on the smells to know, "Is this butter about to burn? So, I better 
lower the heat." Or "Is the rice bubbling too much? Better lower." I’m monitoring those 
activities with many senses. And some of it, we become completely unaware of. We just 
respond. The way walking, right? Walking or running. We're not aware of all the movements. 
Or typing. Once we had to be aware, but by now we don't, it's automatic. And there are 


benefits and costs to that, as well. 


Frode Hegland: Yeah, I'll hand it over to Luc in a second. But I just wanted to say, I think 
that interaction was really nice to hear because, for so many decades, we have had this 
nonsense that interaction should be invisible. They should absolutely not be invisible, 
depending on when you need them. If you're walking on the ground, as you said, even with 
shoes you can tell what kind of ground you're walking on. That is really useful, especially 
now in winter, when it may be icy. So please, let's highlight how we use our bodies and 


interactions. That was wonderful. Luc, please go ahead. 


Luc Beaudoin: Hi, Barbara. I’ve got a number of background projects. They're just 
background projects in spatial cognition. I’m associated with Aaron Sloman in Britain. I don't 
know if you know him. He has a project on spatial cognition, the evolution of spatial 
cognition from an AI. Aaron Sloman, you know him? There are two Aaron Sloman, one is the 
psychology guy, and the other one is the philosopher. 

Barbara Tversky: No, I... The psychology guy was a student. 

Luc Beaudoin: No, this one is technically a philosopher, but he is an AI person. But anyway, 
I'll jump to something that's not with Aaron’s project, but another interest of mine is 
mnemonics. I’ve been doing visual-spatial mnemonics myself from a scientific perspective, I 
miss the beginning of your talk but I take it you've argued for the primacy of motion. 
Basically, motion coming before language and evolution. And there are various arguments for 
that. So, that makes a lot of sense. I see, as you do, the spatial cognition, spatial and 


movement cognition being fundamental. So, as such, I would think that for mnemonics it 
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would be helpful. So I, myself, when I’m memorizing lists, you know that lists are the 
hardest thing to memorize. But if you can turn them into a visual-spatial sequence. And I’m 
not a dancer, so I’m not very good at the visual-spatial motion thing. But I found that if I can 
use a gestural mnemonic, then I can remember these lists. So I remember, Jordan Peterson 
has these 12 rules in his first book and I thought, Okay. Well, how do I memorize that?" I'll 
turn it into a little bit of a dance and the whole thing came out within two repetitions. It was 
quite powerful. But I haven't actually delved into the science of this. But it's something I 
thought, "Well, if nobody's done this, I want to do it." Are you aware of research on using 
gestures for mnemonics? For remembering? Apart from drawing, I know that there's research 
on drawing, how that helps remember stuff. Actually, I’m more interested in imagined 
gestures, because I don't think you need to do it. We know that in sports, athletes often will 
imagine themselves doing things and that helps them execute the behaviour and practice. So, 


there's your question. Imagine gestural mnemonics. 


Barbara Tversky: So, a visual practice, or visual-spatial practice, visual motor practice for 
divers, golfers, or whatever does help. It helps mainly in sequencing. It doesn't help in the 
fine aspects of the motion. Real practice is better than imagining practice. But imagine 
practice is also effective in the absence of real practice. You can do it on the train. I 
remember, it has happened to me several times, on the New York subway, I see singers with 
scores in front of them, and they're imagining the music. So, the part of the method you're 
describing is one of the oldest in the world. It's the method of loci, that was invented by the 
Greeks, Romans to remember their long orations. They would imagine themselves on a walk 
through the Agora marketplace and put a portion of their oration at each of those places and 
then imagine that. So it links things together in an organized way. You still have to form that 
association between the place in the marketplace, and what you want to remember. The same 
would be true of gestures. When I was learning Latin ages ago, there was a whole set of what 
essentially were cheerleader exercises for remembering "amo, amas, amat," and you could go 
through it for real, or you could go through it gesturally. So, those things can work for some 
people, and it's usually for meaningless information. Meaningful information it's better to link 
through the meaning, but images will work. This famous mnemonist beautiful book by Luria, 
"The Mind of a Mnemonist," he certainly remembered himself going through walks and 
placing images. Again, you could place gestures in the same way. I mean, it can all be 
effective, what works for one person. And people rediscover these mnemonic devices. Every 
10 years, write a book, it's a bestseller, and 10 years later, the field is ripe for it again. Diet 
books tend to come out a lot faster. I think more people are worried about their waistlines 
than their memories. But there are those advice books and they would include motion and 


gestures as well. We've done a number of studies, many on people learning complex material, 
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like in how a car break works, or an environment. And as they're learning, they're reading 
text, they're gesturing. And the gestures are making a model of what they're learning. So, 
they're putting down dots and lines for the descriptions of the environments. And when they 
go to recall, they make those same movements again. So, it's clearly helping them recall as 
well. And if they gesture both at learning and at recall, they remember much better. And these 
are spontaneous, the people aren't even aware, really, that they're gesturing. We don't tell 
them to gesture. The gestures come from their body. Everybody learns them in different 
ways. Gestures, unlike words, aren't decomposable. And you could see that with conductors. 
You go and watch the same concert with different conductors. They're gesturing very 
differently. The orchestra can respond in similar ways. So, that visual-spatial language of the 
conductor can be quite different. We went to the opera two nights ago, the guy was dancing 
up and down and he was a joy to watch. And there's research showing audiences respond 
better to conductors who jump up and down. There's a famous video you can find of Leonard 
Bernstein conducting, I think Mozart, some classical piece, with his eyebrows. He had very 
expressive eyebrows. Nothing but his eyebrows. Now, they were well-practiced. But 
(INDISTINCT) and if you want to watch a really gymnastic conductor, watch him. I haven't 
seen him in years but he was a master. And there were (INDISTINCT) using the motion in 
very complex ways to guide the music. And it makes a huge difference. 

Luc Beaudoin: Okay, can I squeeze in another question? I’ve often thought people who learn 
pictorial languages, or languages with calligraphy, that they would basically have better 
memory for concrete words, as well, because they can actually go through the gesture in their 
head, so it kind of adds to it. Do you know of any research on that? 

Barbara Tversky: There's research on having more than one code for memory. If you have a 
verbal code and a visual code, you're going to be better at remembering something, because 
you have more retrieval cues. And if you add a motor code, which could be gestural, you'll 
have even more. If you get too many you might get confused, and it might be hard to 
construct them. But having more retrieval cues for the same bit of memory does work. So, 
drawing, imagining what something looks like, imagining how you would interact with it, all 
of those things can enhance memory, and there is plenty of research on that. 

Luc Beaudoin: But not specifically on people who know calligraphy, or who do calligraphy? 
Barbara Tversky: Some of that is going to be content-specific. Radiologists, who are trained 
to look for one kind of thing, like breast cancer, might not be good at broken bones. So, some 
of it is going to be quite content-specifically. The particular patterns of pixels that tell you 
that there's cancer, are going to be different from the particular patterns of pixels that are 
going to tell you it's a break. So, the movements for calligraphy are to make characters, they 
aren't to make images of people. Although, plenty of calligraphers could do both. Some of it 
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is going to be content-specific, and some of it is going to be more general. And there you 
need to look at the specifics to know the answer. 

Luc Beaudoin: Thank you very much. It's a pleasure meeting you. I cited you in my 1994 
thesis, I counted four times. 


Barbara Tversky: Okay. Thank you, thank you. 


Brendan Langen: Hi, Barbara. Thanks so much for the talk, this is really neat. And as a 
funny aside, I’ve recommended your book to pretty much all of my friends who've recently 
become parents. I think there's so much in the first few chapters, where you just lay out how 
children learn, and how to create trust. You mentioned some of that. I’m really curious about 
how some of your findings can come to life in some of our software tools? So, there's quite a 
movement going on in some of the knowledge creative tool space. You can think of things 
like "Notion," or maybe "Sigma," or even "Roam" research and other notebooks. What 
opportunities do you see for embodied cognition and spatial thinking in our knowledge tools? 
Barbara Tversky: Oh, so many. And then, they'd be specific. But thank you for the 
recommendation. I keep thinking and saying to publishers, "Somebody needs to write a book 
for new parents, and what to watch for." From new-borns, because until children speak, I 
think parents aren't aware of the huge cognitive leaps that children are... Because they're just 
too subtle. And if you learn what to look for, it adds to the already thrill of having a baby. 
And I don't really have the tools and the background to do that, but other people do. Yeah. I 
think there are so many opportunities for adding visual-spatial and embodied, what your body 
is doing. I mean, gestural interfaces have already done that. They've ruined my thumb. And I 
take pity on the people that have been exercising their thumbs from very young ages, because 
of what's going to happen to your thumb when you get to be my age. And voice interfaces 
may help them, but they have other disadvantages. And sometimes people ask me, I have 
worked with people in HCI, and computer graphics, in AR, VR, and I’m really enthusiastic 
about all those media. Some of the work we did with AR was trying to make people's 
interactions within finding their way in an environment, or repairing, or assembling 
something, as natural as finding your keys and opening a lock. So, there were ways of 
guiding your body to the right place. First, by having a virtual tunnel to guide your body to 
the right place, and then guide your head so that your eyes are looking at the right place, and 
then guide your hand to where you should make the motions. And then, it becomes as natural 
as doing something that you've been doing a thousand times other than doing something new. 
And so, that's one example, but I think there is a huge number, and I’m really excited about 
what are the things you guys can do, and how they can make them more natural and 


comprehensible on the input side to people. Maybe you have thoughts. Because there are 
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specifics you're working on. 

Brendan Langen: Yeah. Well, you just kind of hit on something that might make sense. 
There's been some talk in the chat about these findings for education. And I can almost 
imagine a crossover with a tool like "Figma," a design tool for early-stage designers. And if 
you can guide them through the process, that is helping them create something that's more 
stimulating or sound in its interaction design. I could see that being a huge advance. Really 
curious to keep seeing where this research leads. Thank you so much, I appreciate the time. 
Barbara Tversky: Yeah. Probably in the late 80s, early 90s, there was a Shakespeare scholar 
at Stanford, who was designing something that would stage Shakespeare for students. And 
that was prescient but close to what you're saying. And, yeah. I think you can go a long way. 
One problem is scale. And in there, maybe VR is better because you can get things at scale. I 
mean, same with architecture. But, yeah. Tools that can allow me to imagine things that 
would take forever to create. And therefore, create better. Would be phenomenal, absolutely 
phenomenal. 

Brendan Langen: That's really interesting. Almost like bringing along a "now" sentiment 
into the mix, where something that takes so long to build, is often outside of the reach of 
what we can comprehend. That's really neat. 

Barbara Tversky: And on education, I want to just put in a small plug for some research we 
did with Junior High Science Students. We had them learn molecular bonding, and then half 
were asked to make visual explanations, and half were asked to do the normal thing you do 
on a test. Make or take notes someone raised at verbal explanations. And first, we tested them 
after they learned it, which was several days in the classroom. And the two groups were 
equal, we divided them into two groups. After creating it, all the groups improved without 
new learning. So, the process of making an explanation consolidates the material, and makes 
you question, "How could this have happened for an explanation?" So, both groups do better. 
But the group that made the visual explanation did way better than the group that made a 
verbal explanation. So, this is natural for science, because science is so visual-spatial, 
chemical bonding. But their diagrams were so different. Some had sharks grabbing electrons. 
Some had stick people giving them. They were adorable. And you can do it for history, you 
could do it for a Shakespeare play. What are the relationships of all the characters? What 
happens over time? I discovered my father's old version of "Anna Karenina" and I stole it 
from him many years ago. He didn't mind. The first thing it has is the family tree. He made it 
to understand all the familial relations amongst the characters, and then all their nicknames. 
Because Russians always have tons of nicknames. So that helped me reading it, and he made 
this. My kids doing "Dungeons & Dragons" years ago, the first thing they did was make a 
map. Again, from language. And that helped them with keeping track of where they were 
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going in the game. So, education. Yeah. Creating visual-spatial representations of women's 
drawings is one form, they're easy, they're cheap. But doing it in a computer interface might 
work as well. Sometimes I ask, "What does all the technology add over pencil and paper?" 


And I think it's an important question to ask. 
Brendan Langen: Without a doubt. Well, thank you so much for the exploration there. 


Peter Wasilko: Yes. Do you use any mind mapping tools? And if so, how do you approach 
building a mind map? 

Barbara Tversky: I’m sorry, what was that? How do I put what on a map? 

Frode Hegland: He asked if you use any mind mapping tools and if so, how do you go about 
building a mind map. 

Barbara Tversky: It probably depends on the content. I mean, you're going to start with a 
network of sorts. The trouble with the network is usually that the lines aren't labelled. The 
relationships, you're just labelling that there is an association between "A" and "B," or "B" 
and "C." And you probably want to do something more demanding, and specify what the 
relationship is, and then you can cluster things. But it really, in many ways, depends on the 
content. And you can see those of us who remember learning sentence diagramming, which 
was essentially a mind map, and I loved it. Or logic. You could visualize in one way or 
another. So, to some extent, it depends on the concept. But I think, just making networks, you 
want to go beyond that and talk about what is the nature of the lines. The representations. Are 
they inclusion? What are they? And then, go about grouping them perhaps, clustering them 
along common relations. And then you can go hierarchically like a phylogenetic tree. And 
even a phylogenetic tree has been the basis for a great deal of controversy in biology. Where 
do different creatures belong? Is there another life form? And of course, one eukaryote and 
whatever, it was long after I learned biology. So, that particular way of visualizing really 
helped. Bill Bechtel did great work in an actual laboratory, I think looking at diurnal rhythms. 
And they were diagramming for themselves almost every day what they were finding. How 
did they do uncertainty? This is a big issue and a big question. They put question marks. So, 
they put in relations, the best they knew, and where they didn't know things, there were 
question marks that meant, "That's an open problem, let's look at it." It really depends a great 
deal on content. But certainly, there is research showing that kind of mind mapping helps 


people organize their thinking, and learn, and communicate. 


Frode Hegland: Thank you very much. So, I have a question. And that is based on my 
current passion, or what I think is a realization, but I may be wrong. I feel that, within five 
years, we'll be living a lot inside VR, AR, those kinds of spaces. And that's kind of a subset of 
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the bigger cyberspace. But a lot of this seems to be about being disembodied, walking around 
with an avatar that's like a Lego situation. I know, Brandel, I see you're going crazy there. So, 
my question for you, Barbara, is: How do you see VR with full-body immersion where we 
really use our senses to the full, in the context, not of necessarily social interaction and 
gaming and play, but more in the relationship of work? 

Barbara Tversky: Five years seems to me, very optimistic. Partly because people get 
fatigued in AR situations. I get fatigued. There is an uncertainty about moving around when 
you know you're not really in that space. And so, a lot of that needs to be worked through. 
And like "Zoom," they're going to be advantages and disadvantages. And we'll see them as 
we go. The... I’m blocking on his name at Stanford, the guy doing VR in social situations. 
There are going to be, I mean, we're going to have to do it. There are cross-national teams 
doing design, and you can't fly everybody all the time to be together. So, it's going to happen. 
Yeah. Jeremy Bailenson, who's done wonderful work on social interactions, and those might 
be the most important for people. If we found that the internet was used to send emails to 
friends, children, and other people that we love, that was an early use of a massive. They're 
going to be early uses of VR to be with people we love. And "Zoom" isn't sufficient. I still 
can't have a grandchild sit on my lap and feel the closeness. But I do think they're going to be 
increasing uses, they're going to be difficulties encountered and some of them will be 
overcome. I doubt that we'll all be living in the metaverse, although again, I could be wrong. 
You need to talk to the 20-somethings that are already playing multi-person games. And it is 
a bit of a drug. And Yuval Harari imagines that AI is going to replace huge numbers of 
humans in the way that, the rest of us who are useless will exist as in this metaverse where 
we'll be, and it sounds a little bit, to me, when those people talk about it, like somebody's 
conception of heaven. You can have avatars of all the people you love. But then your 
interaction with them might not be taking place in their metaverse. How do you reconcile 
them? What age will they be? So, there are all kinds of cognitive and engineering ideas that 
need to be worked out. 

Frode Hegland: I’m not going to let you get away that easily, Barbara. And first of all, 
Brandel is up after me, and he has an extensive, deep understanding of a lot of this. But let's 
forget about the "Oculus" and that kind of current stuff. And let's forget about timeline. Let's 
say that we have a future where we can, like the "Holodeck" in "Star Trek," we can go into it, 
whether we're wearing something or not, this is very secondary. But there are two things that 
we can change. The external stuff, the environment, and the things we interact with. But also 
ourselves. So, even though we do take advantage of all this VR, with our movable hands, a 
movable head, and all of that good stuff. With your deep knowledge of the human body and 
the human mind, and completely free of technical constraints, being completely fantasy, what 
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kind of situations, or opportunities, or issues do you see for how we work together on 


important problems? 


Barbara Tversky: First you talked about the individual, then interpersonal. As an individual, 
I could imagine situations, interactions, environments, objects I’m trying to create. I can 
imagine them. But until I put them in the world in some way, my imagination isn't complete. 
And this is why designers draw. They can't hold the whole thing in their head. So, they put it 
down with tokens or a VR in the world. And that gives you feedback. It makes you see 
things. It expands the mind in ways that your mind can't do. So, that power of technology is 
awesome as ways of expanding the mind, so that I can create better fiction, better buildings, 
better interactions with people. I can imagine role-play. So taking the things that we already 
use for augmenting our imagination, like role-playing, like creating prototypes, scripts, stage 
designs, whatever it is, and turning them into technology, and making it easy to do those 
things, and explore them, could be awesome. In molecules, combining them in just games. A 
deep mind has changed the game of Chess and the game of Go. People are now interacting 
with those machines, studying the games that AlphaGo can do. So, I think that is mind- 
blowing, absolutely mind-blowing. The social interactions, I don't know how much we want 
to replace them. Now, there are times when I wish I had interacted with somebody differently. 
But I can't redo it. I can redo it in my mind, but I can't redo it for real. So, the social 
interactions, it seems to me, have to be in real-time. Space, we can change. We can all go to 
Machu Picchu together. Explore it together. Enjoy it together. But we can't replay and 
redesign. If I had an avatar of someone I’m interacting with, and I could interact with that 
avatar in different ways, and try out different things, that might help me in my interactions in 
the future. But I can't replay a real interaction in the way that I can replay a fiction. So, am I 


getting closer to what... 


Frode Hegland: It's wonderful, and very deep what you had to say. Very unexpected, which 


is, of course, what I was hoping for. Thank you very much. 


Brandel Zachernuk: I’m trying to decide which of the two questions I want to ask. I’d love 
to get you to go to both but I’ll start with just one. Have you done any work on the cognitive 
differences between writing script with a pen, versus typing, versus dictation for the purpose 
of producing text? What sort of internal cognitive impact there is in any distinctions that you 
would draw? Or do you see them as equivalent? 

Barbara Tversky: Again, I would think it would depend on the person's adeptness with each 
of those and the content. One of my former students, Danny Oppenheimer, who does very 
innovative research, tried to show that taking notes in classes with a computer wasn't worse 


than writing. And the work didn't replicate. Unfortunately, that happens to a great deal of our 
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research, and I think the failure to replicate means, probably, it works sometimes for some 
people, and it isn't a general phenomenon. But what I thought, at the time, is when you write 
it takes more time, so it makes you summarize. And when you type, the temptation to type 
down words in a row way is probably not the best way of learning. You want to wait, 
summarize, write down little telegraphic notes. And the other thing that writing allows you to 
do is array them in space conceptually. In that sense, I think that could help, but it depends, 
really, on what you want to learn. So, as a learning tool, the only research I know of is Danny 
Oppenheimer’s, and he did find writing was better than typing on a computer. And there, I 
think, it really does have to do with how you attend to the lecture. But that work didn't quite 
replicate. But I have a feeling that those... I’m now in an ed school, I was in a psych 
department where you try to get the minimal features that are accounting for something, and 
in ed school, you throw the whole kitchen sink at something and you don't care about what 
works. But nevertheless, people are asked, Are animations good? Is writing good versus 
typing? And people want a blanket answer, and then we say, "It depends." And people don't 
like that answer. But I’m afraid that is probably closer to the truth. I mean, we're living in a 
Covid world now, and it's how do you give advice, and when the target keeps changing, and 
the disease keeps changing, and people are left with the old ones, and then complaining they 
can't give coherent, clear advice. So then, they toss everything out. Which is the wrong thing 
too, because there is good advice, it just keeps changing. 

Brandel Zachernuk: Douglas Engelbart had a famous thought experiment of attaching a 
pencil to a brick and calling that a "de-augmentation" because of how much more difficult it 
would be to write with a whole brick on a pencil. But it occurs to me that, while it would be 
definitely slower, the words that you would tend to write, as a consequence, would be 
significantly more momentous and important for you. Only because you remember the effort 
that would be expended in it. 

Barbara Tversky: Right. Any learning method depends on that. How much are you putting 
into it to learn it? And you're going to put different things in depending on how you're going 
to be tested. How you're going to use the information? How you're going to retrieve it? So, 
you want your encoding to anticipate your retrieval. What information are you going to need 
and when? And that's a more subtle set of considerations. I’m afraid I’m exhausting people. 
Frode Hegland: Quite the opposite. I have two questions. But first, I’d like to ask, we have a 


few new people here today, Karin and Lorenzo. Have you got any questions or comments? 


Karin Hibma: I am just typing my goodbye now. This was brilliant, Barbara. Thank you so 
much. And thank you for the invite, Frode. I am a name or a language creator, and I’m always 


thinking forward. So, it really helps me to understand the antecedence of these kinds of 
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understandings. And I love the aspect of mapping as a place locator for putting words 


together. And thank you. I am still absorbing. So, really brilliant. 


Frode Hegland: Karin, you said you are a language creator. First of all, I obviously 
pronounce your name completely wrong. What is your preferred way of saying your first 
name? 

Karin Hibma: I’m Karin Hibma. People get the Himba, and there's a tribe in Africa. But 
that's not me, as you can. Hibma is a region in Northern Netherlands, a lot of last names with 
"ma's" in then. I think probably means "by the ocean," "by the sea." But everything in the 
Netherlands is. I’m responsible, with my husband who's deceased now, for naming "Kindle" 
and "TiVo" and a few other little things in the world. And I work with companies doing 
strategic identities. So, a lot of times we're either creating names for new products or helping 
them define their language and their story, to get from where they are, to where they want to 
be. Which, of course, goes with (INDISTINCT) and the wonderful concepts you've done. So, 
I don't have your book, but I’m certainly going to be getting it and studying it to cover the 
cover. And the "Babies Build Toddler's" book that I mentioned 1s really brilliant. It's a 
Montessori method, but very often, as I think Brendan said, "New parents don't really 
understand the math." I mean, they're suddenly given this human being, which we don't 
realize is going to come to its full awareness over a period of 25 years. And really being able 
to have some kind of guide rails for parents to be able to actualize that, is pretty wonderful. 
So, thank you. 

Frode Hegland: Karin, I have to ask you with that amazing background, if you would like to 
consider writing a piece for The Future of Text Volume II coming out this year? 

Karin Hibma: I would love to. I am the worst writer, Frode. I like to interact, but I find, 
sometimes, putting words down... But send me a note at karin@cronan.com. 

Frode Hegland: Yeah, we met through "Twitter." Thank you. We met through "Twitter" so 
we'll continue there. But what you say there's very interesting because Barbara was talking, 
just a few minutes ago, about writing in space. Yes, that's something really worth drawing 
out, because, in one sense, that's not really true, unless you're writing on sand or a huge piece 
of paper. Because writing, very much, is linearizing. A sentence has to be linear to have 
grammar. And, of course, with software, you can write a little bit here, a little bit there. But 
then, at some point, you have to, and I just finished my PhD thesis, and the hardest bit was 
not writing, that's easy, but kind of blocking it into a thing is impossible. So, I’m wondering 
if Barbara has any advice for all of us, including Karin, maybe in how to consider this? And 
by the way, Karin, for the book, don't be intimidated with how you write. Please consider 
looking at the previous two volumes, it's all over the place, which is a good thing. Anyway, 


Barbara, any thoughts on that? 
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Barbara Tversky: Say what that refers to again? 


Frode Hegland: Yeah. What I’m referring to is, when we talk about text, there is this kind of 
idealized notion that you can write it down in space. But unless you're working in a free-form 
mind mapping software, you're not writing it in space as such, you're writing it in a line. It is 
one single line. It happens to wrap, but it is still a linear line. And in our community here, we 
are trying to do many things with that. Putting it here, putting it there. I see Bob's put his 
camera back on because this is, obviously, very much his field too. But from your work, and 
your understanding, Barbara, can you talk a little bit about, how we should be writing in 
space in an ideal environment? 

Barbara Tversky: There's the writing for yourself when you're working through the ideas, 
and that should correspond to your ideas. Then you have to put it in a linear form for other 
people to understand, and organize it in a way that other people can understand it. If you want 
to communicate directly, like give directions for getting from my house to your house, or 
understanding how molecular bonding works and thereon. And there, one of the principles of 
InfoViz of giving a context, and then the details do go for text. And we found that a little bit 
in some of those experiments, where we go back and forth between a depiction and a 
description, that you want to give an overview, and then, fill it in in some systematic way. 
And the systematic way should be conformed to somebody else's conception to make it clear. 
But that's for writing clear prose. If you want to do poetry or art in drawings, then you're free 
to go all over the place. And that ambiguity and openness allow many interpretations. And the 
ambiguity is what makes it beautiful. It's what makes you come back to it, and come back to 
it. Because you see new things in the same painting or the same poem. Because you're 
bringing things from you back into it and that's a bit of the interactivity that people like and 
talk about in music, in art, even walking the city, you're seeing new things, because you can't 
completely structure it. And that adds. But if you really want people to grasp scientific, or 
historical, or arguments in law, then you have to be more systematic in getting in a way that 
people will understand it. And creating a context, and then relating the details back to the 
context it's a general principle that goes for good writing and good diagrams at the same time. 


So, does that get it your question a little bit? 


Frode Hegland: It really does, despite being distracted by Edgar, who just came here. Do 


you want to say hi? 
Edgar Hegland: Hi. 


Frode Hegland: So, Edgar is four and a half, and he's learning reading and writing in school. 


And to watch that process is endlessly fascinating. It's exciting. 


Barbara Tversky: Yeah. Endlessly fascinating. When you think about it, reading is a cultural 
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artifact. Cultural inventive. And one interesting fact in the brain and letters is, many letters, 
say in English, a small "B" and a small "D" are distinguished by their mirror images. And the 
visual cortex for recognizing figures, objects, whatever object like things, has many different 
parts to it that do slightly different computations. There's only one tiny area that is receptive 
to mirror images. Otherwise, the visual cortex ignores mirror images. So, flipping faces 
doesn't matter, same person. And for many objects, that's true. Letters depend on which way 
they're facing. And every culture, even cultures that read idiographic languages, like Chinese, 
and Japanese, use that same area of the brain to read. The one that distinguishes mirror 
images. And on branding which, Karin talked about earlier, we have icons. Do you want them 
symmetric? Not symmetric? I mean, they become extremely recognizable. Fonts become 
extremely recognizable. Letters are harder to discriminate. But, as anyone learning a new 
script knows, they can be hard to discriminate. But ideographic letters, faces were graded at 


millions of them. Millions may be an exaggeration, but thousands, certainly. 


Lorenzo Bianchi: My question has been partially answered. It was about writing in space. 
Because, it occurred to me, when I was learning Mandarin, so Chinese characters, what 
happened to me is that, even if I was using an App like "Skritter," where you can actually 
trace the character with your fingers, I noticed that the movement, the range of motion wasn't 
ample enough. So, I started experimenting and I noticed that, if I increased the range of 
motion if I started to use my whole body, instead to trace the person, the character of a 
person, I started to do something like that. It was incredibly more effective. But just for me. I 
don't have any more data about that. So it was that curiosity. Because I’m a student of 
cognitive linguistics. I have an interest in body cognition. And I noticed that. And instead of 
reading and writing the characters, I was just actually living the characters with my whole 
body. It was incredibly more effective. 

Barbara Tversky: Very interesting. And you know, the great calligraphers use their whole 
body. And it's the motions and not what they see. It's really the motions they practice, like the 
piano. And they are large motions. I don't know quite what would happen to them, or anyone, 
when they get to be small hand motions instead of the whole shoulder and upper body. And it 
would be interesting to look at that. And if you ever get to Xi’an, which I highly recommend, 
there's a calligraphy museum that has blocks of granite with calligraphy, mostly ancient. And 
they are just stunning. Stunningly beautiful. Without knowing you or someone that knows the 
characters, they will appreciate it much more. And from my understanding, people who look 
at calligraphy make the body motions. Miniatures of them, this is the mirroring. The mirror 
motor idea. So, when they see the calligraphy, there are feeling in their bodies, the motions 


that it would take to make them. And then your pleasure is enhanced. The same thing happens 
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with dancers. When ballet dancers watch ballet, their motor cortex is more alive than when 
they're watching capoeira. And the opposite happens to capoeira dancers. But when you 
know the motions well, your motor cortex is activated just from the visual motion. There's 
more to say on that, and there's a bit in my book on recognizing. If there's time I can tell that 
story about the point life. But I see there's, at least, one more question. 

This is a former Stanford student who did a rather brilliant work, Maggie Shafar. There 
was a technique that was invented by a Swede, Johansson, in the 70s, of dressing people in 
black, and putting lights on their joints. So then, when you take videos of the people, all you 
see are the joints moving. And if you look at a static display, you can find this on the web, on 
"YouTube," point light. And if you look at static people you can't even recognize that it's a 
person. But once the person starts moving, you can see if it's a male or a female. You can see 
if they're happy or sad. You can see if they're old or young. You can tell that from the body 
motion, from the pattern of lights. It only works for upright, upside down doesn't work. 
Although I bet for gymnasts it would. I don't know. But what Maggie did was take pairs of 
friends, have them come into the lab, and just walk, dance, run, play ping pong, all sorts of 
motions that they would do with the point light. And she had several pairs of friends. And 
then, three months later had them come back into the lab. And look at the point light and 
identify them as, "Are they my friend? A stranger? Or me?" So, they could identify friends 
better than chance. But what was most surprising is they could recognize themselves better 
than friends. Now, they've never seen themselves do these motions. Unless you're a dancer, or 
a gymnast, or a tennis player you don't watch yourself doing these motions. So, they've never 
seen themselves dancing, playing ping pong, and so forth. Yet, they could recognize 
themselves better than their friends whom they had seen doing these things. So, the 
explanation is that, watching it activates your motor system, and it feels right. It's like trying 
on clothes, they fit me. So, you're watching that dancing movement, or the ping pong 
movement, and it's more effective for the more vigorous movements, than just the simple 
ones like walking, that you recognize yourself. Your body is resonating to what you're seeing. 
And when it resonates to you it says, "Yeah, me!" So, that I think is fascinating. How much 
the human motor system or mirror motor system acts to understand the motion of others. And 
we've taken those ideas into understanding action, static pictures, and so forth, so we've taken 
those ideas further. But the basic phenomenon, I think, is fascinating. My guess is, with 
calligraphers would be a similar thing. They could see their own calligraphy. But as far as I 
know, no one's done that. 


Frode Hegland: Edgar just wanted to show he has a real bus ticket. He thought it was worth 
showing to the community today. Thank you. But I have to ask you, just really quickly. Who 
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here has seen the movie "Hero?" The Chinese movie "Hero" with Jet Li? Oh, a good couple 
of hands. If you haven't seen it, you have to see it. Randomly it was playing in Soho when it 
came out, many years ago. I was there with Ted Nelson and my brother said, "We have to see 
this." We sat in the front row. Literally, after two minutes in the intro, they both went to the 
side and said, "Thank you." It is basically about, I love "Hamilton" because it's about 
American being written into existence, "Hero" is about China being written into existence. 
That's the worst summary you could ever imagine. It's the most beautiful movie. If you 
haven't seen it, please do. Brandel? 


Brandel Zachernuk: Thank you. So, the question is a little all over the place, but I’m really 
curious what will you do with it. So, first of all, it occurs to me that, I’m not sure whether it's 
psychologically this is the case, but that there are sort of two motor systems in the sense of 
there being a gross motor system, and a fine motor system. Certainly, the way that I seem to 
sort of marshal my actions reflects that. So, I’m curious as to whether you have research on 
whether, the points of light sort of study is clearly about the gross motor system, people being 
able to understand the movement of large-scale kind of limb alternation I’d be curious 
whether that... 

Frode Hegland: Is he frozen? Or is he just playing with us? 

Barbara Tversky: I know. I think he's frozen. He's somewhere in the cyber space. 

Frode Hegland: At least he's frozen at a very engaged moment. 


Barbara Tversky: Yeah, right. But I can answer the questions, sort of, anyway. And that is, I 
think people when they see handwriting, imagine how it would be written. At some point, 
many years ago, I needed to forge my husband's signature on many documents. He was out of 
the country, and I needed to forge his signature. And I sort of went through the motor 
movements that it would take to make his signature. And he couldn't tell the difference 
between mine and his. So, I don't know of research that's directly looked at fine motor. But 
my guess is that the same phenomenon would happen. I do know that when, this is again, 
years ago, more than 20, a friend was working on a pen whose writing could be recognized 
by a computer. And for English, at least, there were 13 strokes that underlay script writing in 
English. And with those 13 strokes, they could read handwriting, and you could pick it up 
with a pen by where people stopped and started. So even processes that we think of as 
continuous are often truncated. So, my guess is that... So, we missed you, Brandel. You froze 


at some point. But maybe you heard. Maybe I anticipated your question and answered it? 


Brandel Zachernuk: Well, I'll have to go back and watch the "YouTube." But I look 


forward to doing so. The next part of the question that I can't imagine you got to was, in 


127 


linguistics, and in information theory, we have this concept of Levenshtein distance. The 
number of permutations that it requires to move from one word to another word. And to me, 
it occurs that the number of points of difference within a word are the things that make it 
differentiable and distinguishable from another word. The more different something is, the 
lower the amount of information required to distinguish it. In terms of action, what are your 
thoughts on the way that different motions are distinguishable and differentiable in terms of 
their cognitive impact? I’m thinking that when we use computers, it's all the same stuff. You 
were just using a mouse and a keyboard in exactly the same way. So, browsing "Facebook" is 
the same as writing a thesis. At least in so far as the forms of the inputs. Do you see it as 
possible or beneficial to draw some of those activities apart from a physical perspective? 
Even if it results in individual input modalities being less optimal insofar as they then have 


the capacity to be cognitively separated? 


Barbara Tversky: That's again going to be a complicated answer, I think. And even your 
question about language, is that hearing or reading? The distinctions that you have to keep in 
mind. Because my hunch is, they might not be the same. And the Roman alphabet, with some 
variations, is used all over. And that's visual discriminability. Fonts vary. Handwriting varies 
in what's distinctive and what isn't. What's important to one language as distinguishable 
might not be important to another. Hearing would be something else. And their expertise is 
going to matter. And redundancy. One thing Tufte always recommends, he has contradictory 
recommendations, but he likes to eliminate chart junk. But ultimately doing that, eliminates 
redundancy. And we need redundancy to understand. Because we're going to be missing 
things. And have redundancy is an error correction in part. On the visual side, similarly, what 
I need to watch a football game is minimal. What other people need to watch it is, again, 
going to be varied on the motor side. And same with dance, or music. I go to the opera a lot, 
and I love it. But my sophistication is at a kindergarten level. There are things I like and don't 
like. And I rely on critics to tell me what to watch, what to attend to, to distinguish one 
singer's... So, a lot of that is going to depend on my expertise. How much I can distinguish? A 
radiologist, we talked about that earlier, they're going to see things in clouds or in points on 
an image that the rest of us won't be seeing. And you need a lot of training to see. So, I don't 


know if that completely addresses your question, but. 


Brandel Zachernuk: | think it's excellent context, thank you. 


Aaron Sloman: Well, since you asked. This conversation has reminded me of a strange 
experience I had many years ago. I always liked music, and at one point, I did play the piano, 
and not very well, then I learned to play the flute somewhat better. And then, I started trying 
to play the string quartets with friends, using a flute to play the violin. Which didn't work 
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very well, but I then, thought I should learn to play the violin. And I really struggled. And I 
remember on one occasion when I was trying to get the kind of tone quality that I knew, my 
wife could get out of the violin, I couldn't do it at all. I put it down and I started watching a 
television program, in which, the Israeli violinist Itzhak Perlman was playing something, and 
I felt as if something had changed in me. It was a very peculiar experience. And the next time 
I picked up my violin I could do vibrato. And I’ve never heard anybody else reporting a 
similar experience. And I have no idea whether any neuroscientist has any idea how that 


works. But it seems to be relevant to what you've just been talking about. 


Barbara Tversky: Yeah. And I’ve had that experience as well as a small child. I skated a lot 
without any lessons at all, and watched people twirl, and couldn't do it, and couldn't do it. 
And then I learned what you need to do, and it was a state change of competence. And I agree 
that sort of thing happens. And a good coach will often use metaphors to get you to do that. 
Telling you, for a tennis serve, how to hold the racket and how to swing. You have to have a 
metaphor for it. And the right coach, or right music teacher, or even the right artist, the art 
teacher will give you the right metaphors to set you up to do the set of actions properly. And 
again, it is that cycle of listening, and doing, and listening, and doing that I talked about 
earlier with the artist. That is a conversation of the eye, and the hand, and the page. So, for 
music, it would be your ears and your hands. And that cycle. And then, you could have, all of 
a sudden, this insight that you often can't articulate. That changes the whole frame of 
reference. 

Aaron Sloman: I felt it was not my eyes and hand, but some deep ancient part of my brain 
that I hadn't been using, suddenly got turned on by watching paramount in a way that I don't 
think anything else could have changed me, not in that space of time. It was a matter of just 
seconds and then I felt different, and the next time I picked up the violin, I knew I was 
different. 

Barbara Tversky: Well, presumably you saw his arms hands bowing, or? 

Aaron Sloman: Yes, I saw something. It was very abstract. I mean I could try to imitate the 
hands and I wouldn't be able to do that. But there was something else about both, what he 
was doing, and also the sounds that were coming out, which together, drove something in me. 
But I may just have misremembered, or misdescribed, and I’ve never had any other 
experience like it. 

Barbara Tversky: You know what I have, and some of how you learn a new language, and 
how to pronounce words, "R's" are always a problem in different languages and all of a 
sudden getting the insight in how to make that sound that you've been hearing. And I’m not 
an adept linguist at all, but there, when I go to a country where, at least once I knew the 


language, I just listen to it. Il turn on the radio and just listen to the sounds and that helps 
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me go back to that way, "maybe I can do it," to make it sound that way. And there I think 
some of it is the motor resonance. From the seeing or the hearing, it transforms into motions 
of your body, in one way or another. But you're absolutely right. It needs to be studied. It 


really needs to be studied. Yeah. 


Aaron Sloman: And it has to make a permanent change in the brain. What that change is? I 
don't know. 

Barbara Tversky: Yeah, I wonder if you go back to the violin. I go back to try gymnastics. 
That was effortless when I was a kid. The muscles aren't strong anymore. The joints don't 
work. Better not. 

Aaron Sloman: Semi-permanent, I should have said. 

Frode Hegland: So, Aaron. I just did the thing of looking you up on "Wikipedia." So, 
obviously from your voice, it's easy to tell that you're from the same island where we're 
sitting. I’m in Wimbledon. And I’m wondering, first of all, how you came across our 
presentation today, our meeting? And also, if you might have perspectives around the notion 
of The Future of Text, which is tangentially and deeply what Barbara has been talking about 
today? 

Aaron Sloman: I’m in Birmingham, in the United Kingdom. I was born in Southern Africa, 
in a little town called Kwekwe, in what was then Southern Rhodesia. And then I had a lot of 
my education in Cape Town, because my parents were misinformed by a teacher. They 
persuaded my parents that I’d get a better education in South Africa than I would in 
Rhodesia. I later discovered, when I had fellow students who'd done their A levels in 
Rhodesia, that they knew all sorts of things and had competencies that I didn't. So, it was a 
struggle to catch up with them. But anyway. So, I had a collection of different backgrounds. I 
came to the UK in 1957. I was going to do mathematics, but I had got interested in 
philosophy, and then I discovered that most philosophers said things about mathematics that I 
thought was wrong. I thought wrong and I read that Immanuel Kant said something that I 
thought was right. So, I switched to philosophy to defend Kant. And I’m still trying to defend 
what Kant was saying in 1781 or thereabouts about the nature of mathematical discovery, 
which has to do with being able to see possibilities and impossibilities in structures and 
processes. Which is totally different from what's currently going on in AI systems with neural 
nets. Where they collect lots of statistics, and then, derive probabilities. And you can never 
get an impossibility out of that. You can just get more probabilities. So, you're asking me to 
say something about where I’m coming from, and what I’m doing, and that gives you some 
of a feel for it. And I now feel that there's a whole lot going on in different disciplines, in 
various branches of biochemistry, microbiology, and developmental biology, which I’m 


trying to put together in my head in a way that will enable me to explain, first of all, how 
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something in an egg can produce a bird that has all sorts of competences that it hasn't 
learned? Like they can go and pick for food and then paddle in the water and other things. 
But not only birds but there are also all kinds of things that go on in eggs of different sorts, 
which produce different sorts of competencies. So I’m trying to see if I can assemble enough 
information from different sources to explain how that works. Because, at the moment, I don't 
think anybody knows it. I don't think anybody understands it. I don't think I will be able to 
explain it. But I might inspire some of the very bright younger people, who are working in 
different sub-fields, to talk to each other, and come up with the new senses as they'll answer 
my questions. That's what I’m hoping for. Sorry, that goes a long way. Well, it's partly related 
to this because I thought there might be something relevant in this. But I couldn't get here in 


time. But at the end, I think, what you were talking about is relevant. 


Frode Hegland: Yeah. So, thank you, Aaron, very happy to have you here. So, this talk will, 
of course, go up on "YouTube," depending on my Wimbledon internet access speed. And we 
will also have a fellow do the transcript. A human, who is very good. He'll make sure he gets 
our names and all that good stuff. Barbara, do I also have your permission to do screenshots 
of your slides interspersed in the transcripts? 

Barbara Tversky: Yeah, it's okay. My caveat is, I’ve been swiping slides from all kinds of 
sources for 25 years and I no longer know even where I’ve swiped them from. And I worry 
about that. I obviously don't have copyright. And my understanding is, it's okay to post things 
that have no copyright. But I’m not absolutely sure. So, that's my only concern. And that said, 
there are plenty of "YouTube" recordings of my slides in different situations. 

Frode Hegland: Yeah, no. That sounds fine. And that's an interesting question. I mean, the 
journal we publish is non-profit, and all of that good stuff, or completely open access. So, if 
someone has a problem with it, that's not a problem. We take it back. So, thank you for that. 
Barbara Tversky: Yeah, I know. When I wrote the book, I had about four times more 
images than my publishers would let me use. So many I got Wiki creative comments. But 
even then, there were doubts and so forth. And I was dismayed when the Metropolitan and 
other museums released all their images without any demand to copyright, only a tribute or 
no payments. And that was too late for me because I wanted to use, instead of quotes, I 
wanted a depiction at each chapter. I’m glad to see, at least, some places are releasing 
copyright. 

Frode Hegland: That's very good. I’m just going to post them in the chat here as we wind 
down. futuretextlab.info, that's where we will be putting all this data. And this is where we 
carry on our dialogue. Now that it's been 2 hours and 20 minutes, which is quite poetic in 


terms of numbers, I’d just like to say, thank you, Barbara. Thank you, everyone, who was still 
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here. Thank you, everyone, who was here earlier. And thank you, everyone, who will be 
listening in the future. And I hope we can continue the discussion. You're all invited to our 
general weekly meetings, as well as of course, our forthcoming special monthly sessions. 
Which I hope will be even a sliver, as good as today, in order to be successful. So, thanks 
very much and have a wonderful weekend everyone. 

Barbara Tversky: And thank you for your excellent questions and thoughts, it was 


a pleasure. 


Frode Hegland: Yeah, it was a wonderful group. All right, take good care. Bye. 
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Bjorn Borud 


Time, speed and distance 


..or “why we’re going to have to talk to each other and not bet on aliens for interesting 


conversations. 


A few weeks ago I had a conversation with someone who was convinced that within our 
lifetime we will speak to aliens. I pointed out that while I certainly wish that he is right, if 
you start to do some napkin math the numbers tend to suggest that this is never going to 
happen. The likelihood is so close to zero that, for all practical purposes, you can assume it is 


Zero. 


I was reminded of this conversation when Frode sent me a video showing what the 
speed of light looks like at the surface of the earth. A video of one circumnavigation of the 
globe at light speed. 


https://youtu.be/1 BTxxJr8awQ 


To our senses, the globe is huge. Even just travelling from Europe to Asia or to the US drives 
this point home. You are hurled around the globe in a winged tube at speeds that are not that 
far from supersonic - and still it takes forever to get anywhere. Amsterdam to Tokyo takes 
about 13 hours. Amsterdam to New York is almost 9 hours. 

At the speed of light you can circumnavigate the equator 7.5 times in one second. To 


our intuition of the physical world the speed of light is immense. 
Computers and light speed 


We are confronted with the fact that the speed of light isn’t particularly fast in our everyday 
life through computers. The most useful time-scale, if you are working with computers, is 
nanoseconds. For instance an integer division on an Apple M1 CPU is about 0.624 
nanoseconds. The piece of code I work on right now can, according to my benchmarks, do 


one unit of work in about 166ns. 


During one nanosecond, light travels about 0.3 meters (in vacuum). Or roughly 
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one foot. Which means that by the time my program has executed that one unit of the 
operation that I was measuring, light won’t even make it across the street to my neighbor. 
Imagine how much work my computer gets done in the time it would take light to travel from 


here in Trondheim, to New York, and back again. 


Jeff Dean at Google used to maintain a list of “numbers every engineer should know”. 
This list tells you roughly what timescale things happen at. There is a website that not only 
shows these numbers in relation to each other, but also shows how these numbers have 


changed over the last 27 years. 


https://colin-scott.github.io/personal_website/research/interactive_latency.html 


Notice the how intercontinental packet roundtrip times have been almost constant over time. 


In cases that are dominated by distance, physics dictate the limits. 


To be fair, there are things we can do about intercontinental packet travel. It turns out 
that the speed of light in a fiber optic cable isn’t c (the speed of light in vacuum), but about 
2/3 c. With satelites in Low Earth Orbit using laser interconnect in mostly vacuum, we can 
probably get the time to traverse the globe down a bit. But there is a hard stop at c. If we’re 
going to communicate faster we need things that only exist in somewhat exotic physics. And 


even then it would be “fiddly” to put it carefully. 

There is a video that shows the speed of light when travelling from the sun and passing 
the planets of our solar system. This really drives home the scale of our solar system. 
https://youtu.be/2BmXK1eRo0Q 

It takes about 8 minutes and 20 seconds before we pass earth. At around 43 minutes we 
pass Jupiter, and as the video ends at 44 minutes and a bit it is still over half an hour until we 
pass Saturn. 

Voyager | has just managed to back out of our driveway. It is at present roughly 22 
light hours away from earth. Which gives us the opportunity to talk about another limiting 


factor. 


Signal strength and distance 


Communicating over distances with the kinds of technologies we use usually implies using 
some form of electromagnetic radiation. From radio waves, through the visual spectrum to 


higher frequencies such as gamma radiation. 


The signal strength of an electromagnetic carrier decreases by the square of the distance 
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between sender and receiver. So when you move 4 kilometers from your house, the signal 
strength is roughly proportional to 1/16 of the original signal strength. 

Remember Voyager 1, the little spacecraft that could and which has now managed to 
make it down our driveway and past the heliopause at the edge of our solar system? Voyager 
1 has a radio that transmits at about 23 watts of power. By the time its radio signal reaches us, 
there isn’t much signal strength left. The signal is on the order of one attowatt - or 10-18 


watts due to the distance it has to travel. 


A mosquito buzzing in front of your face at a Rammstein concert is going to be very 
loud compared to the signal we get from Voyager |. So in terms of our senses, this is very 
hard to fathom. Voyager | is a very faint whisper in the universe - set to a background of a lot 
of local noise. 

On wikipedia there is a page called “List of nearest terrestrial exoplanet candidates” 
with distances given in light years: https://en.wikipedia.org/wiki/ 
List_of_nearest_terrestrial_exoplanet_candidates 

We know that we’re capable of picking up a signal that is on the order of an attowatt. 
We know this because we have received signals from Voyager 1. We can probably detect 


weaker signals, but this becomes tricky. 


The Drake equation 


The second to last piece of the picture that really drives home the reality that while we 
probably aren’t alone in the universe, we will probably never speak to anyone else is the 
Drake Equation. 

The Drake Equation is described as “[...] a probabilistic argument used to estimate the 
number of active, communicative extraterrestrial civilizations in the Milky Way Galaxy”. It 
lists a bunch of factors which it then multiplies together to arrive at an estimate. The problem 
is that even the intervals of these factors span vast value ranges. Have a look at the Wikipedia 
page for the equation to get an idea: https://en.wikipedia.org/wiki/Drake_equation 

Note that it only talks about our own Galaxy. The Hubble space telescope revealed 
about 5500 galaxies over an area that took up just one 32 millionth of the sky. Today’s 


estimates suggest there are about two trillion galaxies in the observable universe. 


But of course, the distances from “here” to “there” are so great that they aren’t even 


relevant candidates. 
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Our civilization 


Homo sapiens sapiens hasn’t been around for all that long. About 160.000 years. As hominids 
go, we haven’t been around for all that long. The fossil record for Homo Erectus suggests she 
was minding her own business for around 1.5 million years before disappearing. 

We have about another 1.3 million years before we make a dent in that record - give or 
take. 

On the other hand, we have figured out multiple ways of not only causing our own 


extinction, but taking everything else with us in the fall. So there’s that. 


So where does this leave us? Well, we’re not going to be talking to aliens. We might at 
some point hear squaks somewhere in the electromagnetic spectrum that could be indicative 
of intelligent life, but by the time we discover it and get around to responding, it is unlikely 
the’ll even be there anymore. 

And we certainly aren’t going to pay them a visit unless we figure out a way to 
download our consciousness and somehow transmit it somewhere else — which is dubious at 
best. Perhaps we can create some artificial representation of ourselves. 

We don’t have to get into the physics of transporting a useful amount of mass a useful 
distance across the universe to say hello, but let’s just take it as read that the numbers aren’t 
with us on that. We’re thoroughly stuck here. 

And in all likelihood, long before talking to aliens may even becomes a real 
opportunity, we’re likely to wipe ourselves out. Which means the only interesting 
conversations we’re going to have are right here. On this pale blue dot. In whatever brief 


moments we have left before someone pushes the wrong button. 
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Bob Horn 


Information Murals for Virtual Reality 


I have been helping International task forces address with big challenges facing us today (e.g. 
climate change, sustainability, etc.) by creating large 5 x12 information murals. Some of 
these murals have been ported into virtual reality as examples of the complexity VR might be 
able to help us think better. The text used on these info-murals appears in small chunks that 
present interesting syntax-semantics problems for us creators and synthesizers. When we can 
solve them, we may be able then to address other difficult issues such as how to manage 
context, how to better portray process diagrammatically, and how to improve our scaffoldings 
for thinking. 


Introduction: my recent work 


For the past 20 some years I have been helping International task forces address some of the 
biggest challenges humanity faces today including global climate change sustainability, 
energy and resources, various aspects of the nuclear situation. Weapons and waste disposal 


good management. 


My role as synthesizer 


My role has been that of a synthesizer, integrating the deep analysis and considered 
recommendations — wall size information displays that contain hundreds of textual chunks 


and hundreds of visual elements, icons , images and diagrammatic shapes. 
Examples of Information Murals 


Here is what some of my information murals look like: 
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Mega-Flu Pandemic -....... Worst Case Scenario.. 


Mural 1. Horn, 2022. 


China — Fears and Concerns -A Concept Sketch  v.1.0 


aoenck ON COAL - PRODUCING Cop 
Le, 
DING 
To 


EXTREME POLLUTION from 
coal burning threatens health 
in many large cities 


Mural 2. Horn, 2022. 


Mural 3. Horn, 2022. 
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The Vortex of Mental Illness 
For many, the overall dynamic of well-being spirals downward into “the system" 


Mural 4. Horn, 2022. 


Overwhelmed by complexity? 


I know that some of you will feel overwhelmed by the amount of information contained in 
information mural. That has to do with your expectations (I imagine) as to how fast you 
should be able to grasp what is on one of these murals. Rather it would be best to consider 
stepping back and looking for the big picture and then walking up to them and looking at 
individual bits of detail and how are they related. Understanding a whole mural like one of 
these is like reading a 50 page report. Some of your fast readers and may read them in 10 to 
15 minutes. Others will take 30 minutes or longer. 
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Why am | here at this Symposium? 


The question: What am I doing here at a conference about the future of text that is mostly 
focused on virtual-reality? 
The answer: Information murals: [ got into this work of making information murals with 
the help of a British diplomat who saw my work and said “This will replace all those stacks 
of reports that sit it all the bookshelves In the foreign office which no one ever reads. You 
must come to the Foreign Office and show them what you do. “ He arranged it. And my first 
big public work was with the British Foreign Office explaining their policies on climate 
change to 180 offices around the world. That was in the early 2000s. 

We then went on to work for four British government ministries to investigate on 


climate change policy. 


Text as idea chunks with subheads 


Yes, information murals are visual. But you will see that there is lots of text on them. You 
will see that all of the text on information murals is displayed in small idea chunks that are 
related by space, color, shape, size, and diagrammatic elements. 

One of the major reformulations of text for complex subject matters will be to divide 
much of it into such small idea chunks. You can call them paragraphs if you like, or concept 
blocks, or boxes, or snippets or anything else. 

The small idea chunks on info-murals consist of one to (roughly) 7 to 10 sentences or 
often in tight diagrammatic format, and sometimes in table, chart or graph structures. 

One of the next major tasks in the future of text is to learn how to manage, arrange, 


sequence, and display small idea chunks with informative subheads. 


Benefits of small idea chunks with subheads 


I believe these small idea chunks will eventually replace the long endless scrolls a writing 
that appear in academic papers and many reports in science and commerce. They will save us 
all immense amount s of time by enabling quick scanning and skipping of what we already 
know. They will help us re-use many idea chunks more easily repositioning them in different 
info-murals. 


Why am I here at this conference? — second answer 
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My second answer is the number of the speakers at this conference who are much more 
qualified to talk about virtual-reality and to make advances in it saw some of my information 
murals in a small workshop that Frode leads. 

These VR-makers immediately — that is overnight — enthusiastically put one of my 
information murals into virtual reality. And in the workshop team began an intense 
investigation how the information murals may help us to think better about our major human 
problems using virtual reality. One of the big puzzles was and is: “What is the unit or 
element of an information map that we should attach meta data to?” 

Using info-mural in VR is been very encouraging to me. I have offered to help them in 
any way I can because we have very large problems in front of us as a civilization and as 


humanity. And we may be able to make some advances on them in VR. 


Transition to other offerings 


Okay that’s what I am here. For the rest of the time that I have on this platform I want to 


identify a few of the things that we have begun to discuss about info-mural in VR. 


Assumption: improve human thinking 


First I repeat an assumption that most of us are making. We believe that we must improve 
our thinking methods. We must improve are thinking together in teams and groups and 


communities of different sizes. Einstein is often quoted as saying... “We cannot solve our 


problems with the same thinking we used when we created them.”!? 


What can we do to move toward Einstein’s goal? 


There are some aspects of information mural reasoning that can help us. Here are three ways 


we need to get started on. 


Problem: Show and link context 


One of the difficult problems is how to represent and to link important context to the thinking 
that we are doing and trying to communicate this context to others. There is great possibility 


for helping many kinds of creative thinkers in virtual reality to do this context-representation 
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and linking work. 


Show and link context...in Multiple Dimensions 


Subject 
matter 


What are the major questions asked? 
What are the major ways the subject 
matter is organized” 

What are the major components and 
how do they fit together into larger 
wholes’? 

* Who has an interest in the subject 
and 
why? 


Document Organization 


* Stakeholders of organizahon 
* Issue history 


* Tite of document 
* Table of contents 


* Index * Professional discipline 

* Introduction by author stating * The setting (Lc., purpose of 
purpose/audience presentation) 

+ Title on page itself * Previous work done in this area and 

* Pages preceding and following published or distributed 


* Relation to the rest of the 
organization's 

mission and current niche in the 
organizational ecology 


+ Titles and subheadings 
* Overview of chapter section 


Interactive 


system 

* Where am I? 

« Where have | been’? 

* What can I do next (including 
where can I go next)? 

* If 1 do any of these things, what 
will happen’? 

* What just happened? 

* What is happening now? 

What equipment is being used”, 


Reader 


* How does this fit into my life? 
« What are my objectives? 

* What do I need to know to 
understand this’? 

* What are the relationships, forces, 
tasks, 


and so forth that bear on this issue’? 
* What does that mean for me? 


Scaffolding 


+ What ts the general scatYolding 
of the thinking activity (e.¢. 
debating, observing, 
decision-making.ete?) 

* Can | use a structured thinking 
methodology for some (or most or 
all) of the needed thinking work? 


Mural 5. Horn, 2022. 


Problem: Show process visually 


Generally the best way to show history or future scenarios is to use some form of 
diagrammatic information murals. In the previous volume two a future of the text, I outlined 
a one million diagram project. I’m looking for young leaders and contributors to such a 


project. The diagramming software I have seen is not good enough for such a project. We 
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need a next level of development in this domain. 


Problem: build solid and supportive “scaffoldings for thinking” 


Different kinds of social messes and problems that we face require multiple structured ways 
to represent the various points of view. We have to figure out the semantic and technical 
structuring of this scaffolding. Many of these may eventually be much more effective in 


virtual reality. 


Offer of help 


These are only some of the tasks ahead of us. There are a great many challenges ahead for 
our species. Some of the work by people in this conference will be important. If I can help 


any of your get started or continue working on these issues, please get in touch. Thank you. 
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Bob Stein 


Journal Guest Presentation : 4 July 2022 
https://youtu.be/aWK39a7a6Gs 


Bob Stein: So what I'm going to show you is Brewster Kahle asked me to sort of think 
about how the archive could be more useful and I got him to hire one of my colleagues 
from the Institute for the Future of the book, Dan Wiesel. And ee chatted for a long time and 
started exploring and we ended up someplace that I wasn't expecting, which was that after 40 
years of elaborating linear texts, I think we have finally figured out a way. At least we're 


hinting at what comes next in terms of how people are organizing content and presentations. 


Bob Stein: Whenever I have a new tool, I put Vannevar Bush's as we may think into it. My 
colleague was a literature major, and he fell in love with Emily Dickinson. And he always 
starts with Emily with a favorite poem by Emily Dickinson. And so these are eight versions 
of the exact same poem in the Internet Archive. And these are all operating book reader 
windows from the archive. And you can zoom in and they all work. And this is going to be 


fast. I mean, it's been running through a bunch of these quickly. This is Dan's wife. 


Recorded Kim Beeman: “Hi, I'm Kim Beeman. And I'm going to talk about a few of my 


favorite cookbooks today.” 


Bob Stein: That's introduction she makes. She is a librarian. If I click on one of the 
cookbooks down here below, it opens up. There's another introduction by her. Dan put this 
background image and these are two versions of the of the cookbook that he found in the 
archive. Here we're just showing that. Let me see if I can get with this. Here, we're just 
showing that we can sync up an audio or video with an object. So I'm going to play this. And 
when she gets through a short introduction, the focus is going to shift to the First Amendment 
and then it would shift to the others. 


Computer Voice: “The United States Bill of Rights. The ten original Amendments to the 
Constitution of the United States read for LibriVox dot org. By Andrea Fiore December 27, 
2007. One.” 
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Bob Stein: We were trying to do a demo where we and we were look, I was looking for that 
famous image on of the first four nodes of the Internet, and I couldn't find it at the Internet 
Archive. I'm sure it's there somewhere, but their search is so terrible. But by complete 
accident, found this talk that Alan Kay gave in 1995 at a symposium event. You may have 
been there, and it's really quite a remarkable presentation of the history of computing in the 
sixties. And I was so excited because Alan made it very clear that the ideological basis of 
what was happening in the sixties was quite different than what emerged by the by the mid 
seventies with Microsoft and Bill Gates. And I really wanted just everybody under 50 who's 
working in inventing our digital future. I wanted them to watch this film, but I realized there 
was no way I was going to get anybody under 50 to watch a film by somebody that they had 
never heard of. So breaking it up into chapters and it just there was nothing out there that did 
what we wanted it to do. And so these are just three very short bits at the beginning. If I talk, 
click on the Engelbart section you get I'm sorry. I'm on a slow connection in a hotel in 
Birmingham, but you get Doug Engelbart's Wikipedia page, you get the mother of all demos 
video. 

You get the mother of all demos Wikipedia page, and you get the brilliant, which I'm 
sure all of you have seen Ted Nelson's brilliant eulogy for Engelbart.And then back to spatial 
data management. Voyager published this fantastic video disc that the Architecture Machine 
Group now the Media Lab made, and these were the liner notes for the video disk. But it's all 
of the early sort of greatest hit demos from the architecture machine group. And these are 
these four were sort of four of my favorites. This is the Aspen Movie Map. And if you'll 
recall, there's a point at which you can stop your the joystick, turn to the left and go into a 
building and explore it. Well, several weeks ago, Google showed their immersive map system 
and it only took them 40 years. But now they're showing people going inside of a restaurant 
and exploring it. And I just thought it was sort of perfect to be able to add that to the tapestry, 
because one of the things about tapestries, I think it's important is that the dividing line 
between a reader and a writer is as thin as we can possibly make it. So it's very easy for a 
reader of a tapestry to fork it. And as I did here, I added this video from the Google 
presentation. This is really an art exhibit. In 2000, we put out a tool called Tc3, which was 
our attempt at the time to get as close to HyperCard as we could. And we gave it to an artist 
who made these remarkable books that they don't run anymore, of course. But I had videos 
that I had made of people working through them. And so this is just a bunch of these videos 
and. I it just it plays but it as a curatorial tool to make this presentation and work perfectly. 
(Get this out of the way. Yeah. Yeah.) 


Vint Cerf: You said. Of course it doesn't run any more. Would you tell me what's missing? Is 
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it an operating system and Apple? 

Bob Stein: I mean, it runs on Windows, actually perfectly. It doesn't run on the Mac 
anymore. 

Vint Cerf: Got it. 

Bob Stein: I mean I mean, almost everything that we did in the eighties and nineties, I mean, 
not almost without without exception on the Macintosh, nothing runs anymore. And almost 
without exception, everything does run on windows. 

Vint Cerf: Wow. That's actually quite an impressive observation. 


Bob Stein: It's an amazing thing that that they have kept this stuff going in windows. 


Bob Stein: So this is interesting. These eight windows are different for different hours of the 
day from a particular television station in Russia. And we wanted to show what Russians 
were seeing during the Ukraine war on their home televisions. And these are all so, you 
know, I can zoom in on these and they all play. 

Computer Voice: Actually. Really, I'm not in Ukraine. Um, but now. 

Bob Stein: What's interesting is that fast forward a little bit and this is the Internet Archive 
just released this. It's a visual explorer. These seven windows are seven different television 
stations in Russia. And these are thumbnails captured during the entire day's presentation. 
And any one of them, I can just click on it. 

Here's basic. Running in in a window. 

Vint Cerf: Wow. 

Bob Stein: A programming book for kids on how to program in basic. And I was thinking, 
wow, wouldn't it be fantastic for a teacher to be able to give high school students the 
assignment of I want you to see what what computing what programming was like in 1980. 
So here's the assignment. Here's a place where you can do it and here's an instruction manual 


if you need it. 


Kim Beeman: Now, this is simply a Wayback Machine page. Got to get this out of the way. 
And one of the things we've done is when we when you put a Wayback Machine page into a 
tapestry, it comes with a with aa scrubber at the bottom. So if I want to get a different date 


for this website. 


Vint Cerf: Wow. 


Bob Stein: It's all just here. And which is pretty wonderful. 
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Vint Cerf: This is startlingly fascinating. And I'm I'm assuming something that I want to 
verify. It looks almost as if each window in the tapestry is running as a virtual machine. So I 
have quite a base for different operating systems and different applications running within 
each each window. Is that a correct assumption? 

Bob Stein: You're probably above way above my technical pay grade at the moment. What 
we're I mean, each one of them is is basically iframes. 


Vint Cerf: Okay. 


Bob Stein: So I don't I don't think there's anything conceptually about what you're saying that 
couldn't be true. In other words, could I be running Parallels in a window here if I want? 
Maybe. I suppose I could. 

Vint Cerf: Well, if we could make it work that way, if these were really VMs, then you just 
showed a way of hanging on to old software and old content. 

Bob Stein: | think that I think that's certainly that would be a goal. I mean I mean, it's not 
we're not there. That's not what I'm showing you right now. But I think in terms of getting 
there, absolutely. That's the intent. 


Vint Cerf: That would be nothing short of spectacular. 
Bob Stein: Good. 


Bob Stein: So this is another piece based on the book blog post. And if you'll remember, 
there was this point around 2005 when Jaron wrote this terrible essay about why he hated the 
Wikipedia. And a whole lot of us wrote in response to it. 
https://web.archive.org/web/20200801071657/http://futureofthebook.org/blog/2006/06/08/ 
shirky_and_others_respond_to_1/ 

Bob Stein: And I was looking at this at this blog post, and I was realizing that all this blog 
posts really was was is a an annotated guide to a bunch of Web links. And I thought it would 
be interesting because we could do it in a tapestry of turning it inside out. And instead of just 
functioning, instead of featuring our annotation to a list of web links, why not just put the 
links themselves live into a tapestry? So you have here's Darren's original essay, and all the 
other essays that are referred to are all here now. We think that the tapestries are hinting at 
least a new media type, but in order for it to be a new media type, it has to be portable. It 
can't just sort of live only at the Internet Archive. So what's interesting is that if I add the 
word 'embed' here. It's going to take me to a page. Where... I'm going to change the width 
here. (1792) And then I'm going to grab this HTML and I'm going to go to. This dashboard. 
This is just a WordPress blog that I've got and I'm going to make a new post. And 'demo 
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tapestry'. Demo for future of the book or whatever. I think I got that wrong. But text, future of 
text. There we go.And then I'm going to put in. The custom Html. And. I'm going to go up 
here and preview and a new tab. And it's going to take that tapestry and it's going to embed it 
into. This blog post. And this is all this is all operating. And so at least showing the concept 
of. Of portability. And there's one more thing to show you, which is that. 

Vint Cerf: So in this particular case, what has actually happened, what has been imported 
into the Web page that you just created? 

Bob Stein: So the tapestries, as you see them, are simply a collection of URLs. I frames that 
so that each one of these windows. It calls a URL from the Internet Archive. 


Vint Cerf: Okay. Okay. Wow. I could call it from anywhere but in this case. Exactly. 


Bob Stein: Exactly. And I believe, for example, when the the tapestry that the Ted Nelson 
YouTube video. I don't know that that tapestry actually. I don't think we had to import that 
video into the Internet Archive. I think we're just grabbing for Wikipedia and YouTube both. I 
think we can just grab the URL. So here's the last one. One of the things that is that we're able 
to take a collection, which is what from the Internet archive and imported automatically into 
a tapestry. And this happened to be a collection of Atari magazines. And I was just playing 
around and I imported it. And so these are all active windows, and each one of these is a 
different magazine. And when I saw this, I got really excited because I realized that in some 
ways what was happening was that I was. Let me go back to the. Don't die on me now. Go 
ahead. Just go back. We said that I was. No. Sorry. I hate to screw everything up at the end. 
Anyway, that that this started to feel like going back into the stacks. 

What we have what we've learned with these tapestries at this point is that. Having all 
of these objects operating in the same visual field is way more different than we expected. 
That seems to reduce friction for the reader dramatically. I mean, if you think of something 
like this, that. Oh, it's fine. Let me go back to one of these. Yeah, something like this, where 
instead of having to go somewhere, every time I click on one of these things and come back 
like you do on the web and you. So you have to think all the time, do I want to explore? Is it 
worth clicking on this? How do we get everything visual at once, visible and once starts to 
make a very big difference that makes it makes the reader encourages the reader to explore 
more. And so when I saw the Atari magazines all together, I realized it started to feel like 
being in the stacks again, where all the books are sitting on the shelf and you just sit there and 
you pick them off serendipitously, one after another. And the cost of opening up another book 
is so low compared to what it's been on the Internet. So this is an interesting shift that we're 


seeing. So I'm going to step yeah. 
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Vint Cerf: It's been, actually I think there's something more powerful happening beyond the 
stacks metaphor and that's context preservation. What's what's happening in the tapestry is 
that it is preserving a substantial degree of context for the user. Exactly. That's a strikingly 
powerful notion. I've never seen it illustrated quite so with such facility. This is really 
fascinating. Have you published anything at this point? 

Bob Stein: No. 


Vint Cerf: Wow. There's one other odd coincidence. There is a company which got started 
about a year ago called V Tapestry. Lowercase v. Capital t. It was started by a woman who 
does. Montages in the course of conferences. You have somebody with a giant canvas and 
people are talking and they illustrate what was being said. And so she does these things one 
after the other. Sometimes it could be a dozen or more of these very big canvases reflecting 
what was discussed and with lots of symbolism. She's automated this process, and so 
Tapestry is a company that will take the incoming text of the discussions and generate 
imagery to automate the process. It's quite different from. 


Frode Hegland: Bob, thank you very, very much. Really good to see this. I'm going to go 
back to the other window because that's my notes. You say that it's way more different than 
you expected. And I know that you obviously have experience with VR going way back and 
to different degrees, and I only became converted by Brandel in January. Before that, I'd 
actively stayed away from it because the future of text was a specific focus and then I decided 
to branch out. Now, obviously, what you're working on here would be tremendous to have 
wall size. Bob Horne often joins our community and he is all about murals, as you know. And 
one of the things that was really shocking to me is that Brandel took one of his murals, built a 
little, relatively speaking, Brandel a little app for it where all you can do is stand in a room. 
There's nothing but the mural. A mural is really big, but you pinch to move it away from you 
and move it towards you. So there's no walking so that you don't get sick or anything like that 
and you can move it sideways. That's all you can do. It's just incredibly powerful. Because 
yeah, it's it's almost undescribable how powerful it is considering there's nothing there. So I 
can imagine what you're working on here. First of all, obviously on the wall, but if this was 
even a normal kind of office room, because when you talk about preserving contexts that I 
could imagine that you literally keep one wall for work, one for a specific project, the one in 
front of you for something else. Because everybody talks about this. What I'm saying is 


obvious. aBut what was so amazing to see today is all the aliveness that comes through it. 


Bob Stein: Yeah. You know, I didn't use the phrase that I should have is that tapestries are 


infinite canvases, so they can go on forever. At which point you need some form of zoomable 
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UI. You need to be able to. You need to be able to fly around in there and zoom in on 


something and expand it. 


Frode Hegland: Oh, that was the other thing I wanted to praise that you showed when you 
clicked on a thing it became ‘full screen’. 

Frode Hegland: That is so important. When I worked with someone on the Chinese website 
for the NBA, the American NBA in China. We built a version of hyper worlds where you can 
click on a player's name and you get a little bit of stats and you would click on that and we'd 
go big. She wanted it to be semi-transparent and smaller, and now that's been arguing with 
her. That's when I realized that if you're looking at something, make it big, because that's 
what you're looking at. Make it quick to go small again. But here, you know, you didn't play 
it with a little bit of this and that. I was just so relaxing on the AI. Thank you, Brandel. 


Brandel Zachernuk: Amazing work, really exciting. One question is, if you are browsing 
the same tapestry in multiple windows, is there there would be a facility for synchronizing 
them, more aspects of them. Is that something that you've considered in terms of either the 
maintenance of sort of view state or the or in order to be able to use multiple sort of nominal 
windows, be they real or virtual, to be able to synchronize sort of views over things? 

Bob Stein: Nope. Really interesting, though. I mean, I think that we just sought to answer 
that partially by going back to what Frode said, which is that I think what somebody asked 
me, so how long does this take you to do? And I said, Well, it's either three months or 40 
years. There's nothing technically very interesting yet about what we've done. Right. But it's 
conceptual. I mean, I was showing this to somebody the other day to Howard Besser. I don't 
know if, you know, he's an archivist at NYU. And Howard was Howard was saying, oh, my 
God, this is the stuff that we imagined 40 years ago that we would do someday. And now and 
what's happened is that the Internet has gotten so much more powerful that things that we 
could only imagine back in the pre-Internet days, but we we couldn't do once once the 
Internet took over in terms of electronic sort of expression, we had we had to really reduce 
our sense of what was possible. But now the Internet, the Web has gotten so much more so 
much better that we're suddenly we're able to do things that we forgot we were interested in, 


in a way. 


Brandel Zachernuk: Yeah. Another question that I have for observation is one, obviously, in 
assuming using a user interface, your documents need to sort of withstand a lot of zooming. 
Does that does that direct and guide your sense of which which documents work? Well, you 


know, like you have you have YouTube videos and archive.org videos that have star frames 
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that are or frames that are representative at some level of them, that that can do sort of once 
not all content on the Internet is so well kind of entitled or predisposed to being being able to 
kind of zoom like that deel like there's anything that can be done to help it in terms of having 
having those things be different sizes or or do have you put signposting that is something 
other than the the documents themselves inside these type of stress to support that. 

Bob Stein: Signposting. Yes. I mean, those are I mean, we are able I mean, just in terms of it 
goes back to to Vint's comments. I mean, let me... First of all, let me try this. One of the 
things that were the key thing that got us where we got to was that when we were working on 
the Alan Kay videos and showing and there were all these objects and ideas that we wanted to 
put together, Dan was reading Merlin Sheldrake book on the communication that goes on 
with fungi in the forest. And Dan said well suppose we actually thought for a minute about 
the fact that the objects in the Internet Archive are like trees. They're the nodes. But 
suppose the connections between the trees had had as much, or at least the important 
information that showing the connections between objects is actually crucially 
important. And that was how we really ended up with tapestries and the Brandel. Both of the 
things you've asked, you've you've asked we haven't thought about, you know, we're it's so 
early, but that's why I like showing it to the smart people, because they, you know, they start 


to raise questions that show us where we have to go. 


Frode Hegland: Well, Bob, that's why you need to come back. We're here same time every 
Friday. Every Monday, except for last week when we had our projects. I see your hand, Peter, 
but I just wanted to do my little standing up on a soapbox for a few seconds, because you 
made such a really important point there, Bob, about this is what we dreamt about 40 odd 
years ago and why hasn't it happened? It's not just because the Internet is more powerful and 
computers more powerful, although, of course, that's useful. It's also because you did it. It's 
really important. So in both paying you a compliment and I'm really trying to highlight the 
fact that commercial pressures are one thing what can augment is another thing. And the 
reason our little group here is now 99% focused on VR is because we're going to go into the 
same situation. You know, I feel almost like badly fired, you know, original Mac people and 
all of that stuff. There's an excitement now and that's all nice. It makes me feel very youthful. 
But what I really, really fear is if that if there isn't a user, an academic community that is 
saying the stuff that you are saying, it's like these are the things we can use to augment how 
we work. It's only going to be How can Apple make more money? How can Facebook 
Sushmita make more money? And that's totally fine. There's nothing wrong with commercial 
development, but what you've done, as you said, technically it isn't the miracle. The miracle 
is that you're paid in the effort and you're not making it available. Right. So when it comes to 
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VR, right now, we have this beautiful oh, it's exciting and new. But in a few years, I think 
we're going to be where we've been for the last 30 years in Flatland. You know, there's so 
many things that can be done, but the market forces are so powerfully doing, you know, 
Macintosh pages and Microsoft Office. Where is the Bob Stein innovation going to fit in 
that? Right. So that is why we're fighting and that's why this future of text this year will be 


how can we work? And we are. Over to Peter. 


Peter Wasilko: Okay. I was wondering if you'd given any thought to multi user scenarios so 
that you could be looking at a tapestry on your machine, but have that synched with the 
tapestry on my machine so we could have multiple cursors visible on the screen at the same 
time, and we could have mixed initiative and exploring together. 

Bob Stein: That is certainly something that we imagine we will get to. I mean, what I'm 
showing you today is simply a proof of concept. We are we have to build this on the on the 
on the other hand, what used to take millions of dollars and years, we now are quite confident 
we can do a 1.0 version for in in months for hundreds of thousands of dollars. But it's you 
know, it's it's going to grow. I mean, you know, and who knows? Our version of tapestries 
may not be the one that grows. I think that's you know, it will happen and and it will be multi 


user and it will be collaborative. 


Vint Cerf: It's been again just I had written down the multi user question, Peter, so thank you 
for for asking it. Our experience with multi user documents at Google has been very powerful 
and for small groups of people. If you imagine, however, that a tapestry is broadly available 
to tens of millions of people, you would not want to have state information for 10 million 
people all dicking around with the same document. So you can immediately see the need for 
some kind of data structures that would isolate the behaviour of a group against this 
background tapestry without interfering with other people who might be interacting with the 
tapestry. So it's an interesting challenge because the current implementation, the object 
contains the state information in our implementation of Google Docs. 

Bob Stein: Our assumption is that tapestries there is there's the there's the understanding that 
if you want to be in a tapestry with somebody else, you have to give each other permission. 
And you're in that instance of that tapestry. And if you decide to fork it, you fork it for 
whoever is for yourselves and not for everybody else, obviously. I mean, we we dealt with a 
lot of these questions when we were doing social book where which never didn't, didn't come 
to market. But, but this idea of, of people reading together and annotating something together 
and how you could do that as a group and not screw up other people's experience. Although 
where we went with social book, which I think was important, was that if the if the 


permissions were in place, I might be reading a book with with Vin. And our annotations, if 
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we made them public, would be available to other people as well. They could basically click 
on a community tab and see everybody's comments. But anyway, the social experience of of 
documents, I think, I mean, Google Docs has sort of been by far the most successful example 
of that. But I and anything less than that isn't good enough at this point. 


Frode Hegland: A hugely important underlying thing here is and I'm going to start with Web 
and go backwards is the infrastructure because also one of my I have two fears about the near 
future. One of them is we're going to run out of imagination in terms of the audience, just like 
Doug and other people, that amazing things in the sixties and then eighties and nineties. 
Desktop PC was defined at being specific things. Imagination went out the window. That's 
going to happen to VR. But the other thing that I'm really concerned about is you go into VR 
an environment, you create an artifact, a connected artifact, you go to another environment by 
another vendor and you either can't open it, which would be absolutely insane. Just like a 
word file in the olden days, right? So what I'm saying, you are contributing here is an 
infrastructure for how you thread these things together. So I think that, yes, this is really nice 
to see on a traditional display, but I think that with real support this and of course what we're 
working on with visual mata a few things to allow you to go in, do amazing stuff, whether it's 
2D, 3D, whatever, and then go somewhere else will be so important. I am so scared. I mean, I 
love and adore Brandel. I'm a mac user fanboy and I'm really scared that when Apple comes 
out with their headset, whatever formats they decide are the initial ones, it's going to be 
cemented in reality forever. We need to scream and say these are the useful and open ones. I 
think that's one of the reasons, it's so amazing to see what you're doing because it's not static. 
It's so dynamic. 


Peter Wasilko: Okay. I was wondering if you'd ever seen the Chat Circle's user interface. 
I'm dropping it into the chat now. That was an MIT Media Lab project that dealt very nicely 
with dealing with groups of people interacting in the same space. And it used a. Basically a 
large spatial plane representing each person in a space as a circle. And you could move 
around and you'd be able to hear people who are within a certain radius of your location, but 
you'd also be able to see the circles of people further away twitching. So you could get a 
sense of where there were clusters of people and that overlay that on your system to provide 
an interface for managing the large numbers of people, potentially interacting in the same 
tapestry. So you can sort of think of they'd be in different phases and you'd be able to see that 
there are other people who are out of phase with you and bring yourself into phase with their 


conversations very fluidly. 
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Brandel Zachernuk: I'm curious so about sort of the authoring picture and and more 
broadly, the way in which you feel so based on the sort of the arrangement of the tapestry that 
you have so far. They seem like they're fairly canonical and durable in so far as you would 
you could point to this tapestry or that tapestry. And so that there's a rationale to have them 
existing as a as a as a distinct artifact that is intentionally constructed and delineated so that 
this is the end of the tapestry, this is what it is. And so, one, I'd be really interested in sort of 
the current state of authoring as you sort of have it, as you desire it and all as well, whether 
whether there's room to to pull on the thread, pardon the pun, of that, that continuity of it, you 
know, how intentionally it needs to be created versus what other options exist in that sort of 
space. 

Bob Stein: Well, first of all, I mean, our assumption is that version 1.0 will you'll you'll 
simply be dragging and dropping from a from a folder of of objects onto a onto the field. I 
mean, right now it's clunkier than that, but it will be very simple to drag and drop and 
assemble it as you want. And when you, quote, publish a tapestry, it is frozen. But as I was 
trying to say earlier, that it's very easy for a reader to push a button and, in effect, fork the 
tapestry and either add things or rearrange things as as suits her and can publish or not 
publish, etc.. 

I mean, I think everybody here will understand what I mean when I'm saying this. I 
think that for me, I'm not a programmer. But when we had the prerelease version of 
HyperCard, when it was called Wild Card, and suddenly I was able to hook up a video 
disc player to a computer and I could start to make things that were had value without 
being a programmer. So HyperCard sort of became and then my, my son, who's now an 
engineer at Google, you know, cut his teeth on HyperCard. And so I it killed me when when 
jobs killed that. And Tapestry's in some ways is our attempt to go back to a time when there 
were tools for teachers and students to start to make things that had value and currency. I 
mean, it's ridiculous that we haven't had anything as good as HyperCard in all these years, 
and that's sort of where conceptually I'm starting from. You know, the tapestries need to have 
hyper talk of some sort. You need to be able to have an event statement in a tapestry. We'll get 
there. I but that's I think that's where we're that's where where our focus is at the moment. But 
it's back to your your question statement. Several people have, when I've shown them this, 
gone a direction that in some ways, thankfully, none of you have gone yet. Which is, So can 
can this can this be hooked up to AI in a way that I, I give I give it I give a subject matter and 
the tapestry is automatically built from. And the answer in my mind is always, Yeah, I 


imagine we could do that. But that's sort of not where I'm starting from. 


Frode Hegland: And so Mark and I spent the last week at the hypertext conference and two 
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relevant things came up again and again. It's. Spatial hypertext is one. This is related to that. 
Why hasn't it been invested in. And also the kind of basic programming you're talking about 
now, if you're going to have a proper hypertext environment, you need to be able to have 
clever links that have a little bit of fun and have a little bit of knowledge of previous stuff. So 
to have like a hyper talk thing now is not going against what you said about not being a 
programmer, even though it could obviously sound like that. I think it's really, really crucial 
to enable users to be able to do some basic scripting without having to go whole hog to write 
this much code to initialize before they can decide what they're going to write below. 
Anyway, that's just me. 

Mark Anderson: | was just wondering, looking at the tapestries and seeing that. So you 
showed us a number of interesting sort of set ups there, and some were in a central grids and 
some of them had a bit more of a theme versus a narrative structure to them. Do you do you 
capture the. I'm trying to avoid the word link, but for the intentionality of placing this thing 
alongside that thing. And I say that with there's no hidden question in that. I'm just I'm just 
thinking of the fact that if I put, say two things together within the same tapestry, I'm doing it 
with some intent. And that's worth capturing at some point, both perhaps for me, for my 
future self, or for someone whom I wish to inform by the tapestry making. 

Bob Stein: The best way to answer that, I think, is that one thing that's driven all of my 
workAll this time has been that when you make an authoring tool that it's important not to 
restrict a single pixel. In other words if Lif I, if I'm really going to empower people to 
make things, then I have to allow them to decide what goes on, what page, where what 
goes into what visual field, where. Because it's a very slippery road. Once you start to restrict 


pixels, you end up in a in a different place. 


Mark Anderson: I just I'm perhaps thinking of I see that. And I called with it. I was thinking 
more than just a sense of understanding that how I when I view your tapestry and understand 
the relationship between the first box and any other box that might be in there. 
Bob Stein: | think that's up to the tapestry maker. In other words, if the connections between 
objects in the tapestry can be made in lots of different ways, it can be made with arrows, it 
can be made with contextual text, it can be made by the placement of two things next to each 
other. I mean, there are so many ways of doing it. And you know, hopefully when tapestries 
come out into the world, whoever does it, there's going to be a lot of exploration at first of 
people discovering new ways to put things together. 

And you know, I'm pretty excited to see what my grandchildren do with tapestries. It 
won't be the same thing. I'm hoping it's far enough away from from linear, from the linearity 


of text that they will get someplace interesting. And I, you know, I, I, I do think we will be 
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most tapestries will be looked at in three dimensional heads, whether it's some X, some form 
of X or, you know, not at first. At first we're going to be on our computer screens. But that 
will change. 

Mark Anderson: Yeah, well, that's good. It's good to hear your point about it, not just being 
a matter of handing it all over to I. Not that's not the iceberg thing per se, but the idea that it 
should be doing everything is is potentially horrifying. So thanks. I'll see to. 


Bob Stein: Well. Thank you, everybody. I really appreciate the opportunity and very there's a 
lot of I looking forward to having the video of this so I can go back and get each one of these 
questions and really think them through. 

Vint Cerf: This is pretty amazing. In an hour or less you managed to essentially upend a 
lot of people's thinking. Mine certainly just one thing which strikes me as being 
extraordinary about this whole design. And it harks back to the basic architecture of the 
World Wide Web. The entire structure that you've described is deeply dependent on reference 
and resolution. In the sense that tapestry is this collection of references and the fact that the 
references have to be resolved opens up this wonderful indirection. Because the resolution 
could change over time. If you had huge demand for something, maybe you turned it into a 
reference later because you couldn't serve up all the video from one website, all those things. 
This is this fact that it is there's indirectness and resolution involved in this. Then the tapestry 
itself is just a collection of references. In fact, it's amazingly powerful when you think about 


the compactness of the tapestry relative to the content that it presents. 


Brandel Zachernuk: I really love... My last question. My my last question is it's hopefully a 
good thing to put a bow on that. So first of all, this is amazing. What's next? And then second 
and related is what do you want from other people, including and most importantly, perhaps 
us? 

Bob Stein: I'm going to think of a good ask. I mean, we're I'm I'm so pleased by your 


collective response. I'd like to think of a really good answer to that question. 


Peter Wasilko: Yes. I just wanted to come out. How much? I liked the observations about the 
need to embed systems so they'd be available and how it's impossible to run old Mac 
software. And putting on my lawyer hat. I think a big problem is everyone is afraid of the 
licensing issues on the core roms for old platforms and Congress could really fix this if they 
just pass a clear bill. It could be a one pager that simply says for purposes of fair use, if the 


rom of an obsolete computing system is not available, copying and reproducing that ROM 
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and making it available to people until such time as the current owner of the IP makes it 
available in a commercial form, shall be deemed fair use and just put that in the law one page 
bill. They can have it worked through in an afternoon and it would solve so much of this 
difficulty. I found wonderful Mac emulator systems, but they would require me to be able to 
boot my old broken Mac that I had a license to the ROM and to be able to get the data on the 
ROM off, which I can't do because the old machine is broken. So even though I'm legally 
licensed, even under the current intellectual property scheme to be able to access that or on 
my physically can't access it and no one is willing to share them on line because they're afraid 
of a lawsuit by Apple or some other mega corp coming after them. And it can be fixed very 


easily. Just declare fair use to reproduce ROMS of obsolete hardware. 
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Brett Jackson 


The evolution of mind maps for interactive VR experiences 


Idea Engine is a virtual reality tool that utilizes mind mapping techniques to convey 
information and narratives, allowing users to create interactive and immersive experiences by 
incorporating text, interactivity, and various display types, ultimately inspiring more people 
to share their stories. 


The evolution of mind maps for interactive VR experiences 


Mind maps are commonly used to express ideas and relationships. Bringing the concept into 
VR felt like a natural progression, providing an abundance of three dimensional space to 
arrange your structures. The more I explored the concept, the more possibilities began to 
unfold. Idea Engine is a work in progress that leans into the strengths of VR and mind 


mapping to convey and experience information and narratives. 


Why text plays an important role 


A page of text affords me the opportunity to absorb information at my own pace, easily 
adding a pause where ideas need time to settle and skimming over less interesting facts. A 
short rewind, when required, is effortless and instant. Ideas and imagery come to mind and 
help to cement new learning as I actively engage with the words, eyes scanning, and attention 
focussed. Text is the most readily available creative tool and can elicit strong mental imagery 


so it was important to enable and encourage its use in Idea Engine. 


Idea Engine 


The aim of Idea Engine is to inspire more people to share their story. It has to be easy to use 
and allow many forms of creativity to be showcased. My focus is on enabling immersive 


experiences that are more interactive than a book and more narrative driven than a game. 


Idea Engine consists of nodes, optionally connected by relationship lines which may be 
visible or hidden. Nodes denoted as children can be set to move with their parents or be 
controlled independently. A node has one or more states with each state having its own 


display type, position, colour, text blocks and interactivity settings (can you grab / take it). 
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The display types are: 3d models, images, icons or text heading. 


Text blocks are attached to nodes and configured to be shown for one or more states. This 
allows text to change in response to user interactions and then be combined into a final 


paginated form. 


Interactivity is the key 

You can quickly surround yourself in VR with visual mind maps using various display types 
and convey complex ideas or scenes, but it’s when you add interactivity that the scene comes 
to life. 


Nodes have a default state and can be initially visible or hidden. You could start with a single 
visible node to draw the user's attention and explain your scenario. We can interact with the 
node in various ways. Just looking at the node can fire an event, as well as touching it with 
your hand, head or another node or by grabbing it. You can add a button for the user to press 
and explicitly fire an event or you can evaluate a formula to fire an event when certain 


conditions are met. 


Events cause high-level scripts to execute. Commonly used functions in a script will play 
audio effects or change a node’s visibility or state, thereby affecting it’s appearance or 
behaviour. I could talk at length about the technical features, but to appreciate the potential of 


the idea it’s probably more efficient to give you a small sample scene. 


Your story 

Gentle music plays while a doorway fades into view in front of you (a node). A button press 
(raises an event) pops up a page of text that describes your environment and the ornate door. 
You’re informed the door is locked. You look around the room (now populated with more 
nodes as a consequence of your button press) and physically walk to a desk and find a key (a 
node). You pick up the key and place it into the keyhole in the door. You hear a click (an 
overlap event has fired, changing the state of the door node) and the textual description of the 


door changes to say it’s unlocked. 


You interact with a few more objects, learning more background to the story. An icon offers 
you the option to climb out of the window, but you choose to grab the handle of the door 


(grab event) and are instantly taken to a new location with a new set of nodes. 
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Highlights 
Importantly, nodes may be revealed as required to control the flow of the narrative. Users 
progress by interacting with the experience, creating their own non-linear path. It’s easy to 


create branched narratives. 


The author can present a hierarchy of related information and the user decides how deeply 
they want to go. Maybe they will seek out every grain of background story or move on more 


quickly when time or their interest is limited. 


Nodes may be placed around the virtual environment, requiring physical movement as though 


being in the scene, adding to a sense of immersion. 


Idea Engine also has variables and inventory systems so you can keep track of what a user 


has achieved or require them to bring an object to overcome a later obstacle. 


Resources 


The scene may be quite abstract, leaving much to the imagination. 3d models may be used for 
hero objects where you want to see and understand every detail. Other objects could be 
described with a pencil sketch or a photo uploaded from the author’s mobile phone with an 
optional button giving a detailed description. Some nodes may just use a textual label or icon. 
It all comes down to what resources the author has, or is able to create, and how they want to 


present their story. 


Sourcing content 


There are thousands of pieces of freely available content online that can be used to build your 
story. You can import 3d models with inbuilt animations, music and audio recordings for 
sound effects, equirectangular images that fully surround you and other images for objects. 
You can use AJ art to turn text into images and help inspire you further. Hundreds of inbuilt 


icons can also be used. 


Rather than kitbashing your project, you may want to present your own talents. You can 
record your voice from within Idea Engine for speech and sound effects or easily import 


photos and text from your phone or computer. 
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VR-specific considerations 
The immersive nature of VR draws the user into the story, giving them a true appreciation of 
scale while detaching them from distractions. Idea Engine allows each location to have its 


own music, background imagery and lighting to help further set the mood. 


We sometimes need to direct the user’s attention in these 360 degree experiences. Spatial 
sounds can help turn a head to face the origin of the sound. Idea Engine can also animate the 


appearance of relationship lines between nodes to guide the user’s gaze. 


Viewing block-text generally occurs after getting close to an object and clicking a button. 
This puts you in an optimal position to read the text with clarity. It’s important to allow users 
to grab the text and move it to a comfortable position to read. Text gradually fades out if you 


move away from it, otherwise.you will soon observe ugly artefacts as it becomes unreadable. 


Bulk text editing is implemented as per a mobile phone, including finger-based highlighting 
cut and paste etc. At the moment, I wouldn’t expect people to type large volumes of text 
while wearing a VR headset. Comfort needs to improve, virtual keyboards only suffice for 
short text entry and physical keyboards can be more difficult to use even if passthrough 
cameras make it possible. I recommend users type the text for their stories on their preferred 
device. They can then connect to Idea Engine via a web browser and simply cut and paste 


text into the appropriate nodes. We don’t have to force everything into VR. 


Laser pointers are not used in Idea Engine. You reach out and interact with the world using 
hand tracking or tracked controllers. I believe that seeing and using your hands in VR 


provides the most immersive and intuitive approach to interactivity. 


There is no point in having a blank back side to a user interface so buttons and text appear on 
whichever side you happen to be looking at. This is an example of divorcing our thinking 


from real-world restrictions and making the most of VR. 


Finally, this can be a shared experience, capitalising on another strength of VR. Maybe you 
will venture down different paths and update each other from afar, or read passages to each 


other and discuss your thoughts. 
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Idea Engine is a tool with many potential applications, from helping users navigate and 
understand complex ideas, building training scenarios to test and evaluate employees, to 
creating interactive games and novels. I hope it inspires creativity and helps to bring 
immersive experiences to life. You can keep up to date with my progress on Idea Engine at 


https://twitter.com/JumbliVR 


action befitting of a 77 year old. 


“Remind me to take my medication in 
15 minutes.” 


The Al assistant replied “Medication 
reminder set for 7:15pm.” 
“Engage.” 


The room vibrated as the sonic haptics 
came online and the visual immetter 
band flickered around the perimeter of 
the room. 


“Load Alfie's Quest 2020.” 


1 node 
4 states 
(Ball) 
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Daveed Benjamin 


Thoughts about Metadata 


I applaud the Editor’s Introduction. Below are some thoughts that I had while reading the 
sections The Future of Us, The Future of Text and Improving not only VR Text or AI Text, but 
ALL Text. I present these thoughts because they add to the conversation and are part of the 
design requirements for the Overweb, a decentralized meta-layer that augments online, 


virtual, and physical realities. 


1) The creator cannot own, be responsible for, or control the metadata for their creation. We 
can’t rely on the creator having the knowledge, capacity, and interest to create or moderate 
metadata for their own work. Different metadata have different sources. Some can be 
automated, such as creator, title, and date. Others can be from the creator, such as the 
creator’s notes and tags. Some need to be the creation of the crowd and/or AI. The opposite 
of this is Today’s Web. 


2) Best practice abstracts metadata creation into a decentralized public space that any 
known persona can contribute to. While we can embed metadata in documents, we can 
also abstract metadata into decentralized storage that bi-directionally links to the 

document. This enables large amounts of metadata, including multiple perspectives 

to connect to but not weigh down the original document. This model facilitates metadata 
creation by others than document creators. But this presupposes a unified metadata model 


across documents and applications. 


3) Metadata can overlay everything (e.g., the Web, virtual worlds, the physical world) and 
be triggered by anything that creates an event (e.g., QR, text, imagery, 3D models, 


sounds). 
4) Anyone can publish (subject to verification), curate, prioritize, and filter metadata. 
And they should duly receive rewards for their contributions. We call this a fair value 


exchange. 


5) Censorship-free environments need effective metadata filtering mechanisms. People 


need the ability to create their own algorithms and thereby choose their own adventure. 
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Personal algorithms should be tunable, transparent, adaptive, and portable. We call these 


smart filters. 


6) People can be pseudo-anonymous. They should benefit from their creations and activities 
and also be accountable for them. This suggests a unified one-account for life decentralized 


identity and security model. This is a non-trivial problem. 


7) If Twitter is the digital town square a la Elon Musk, it needs a digital town library for 
the metadata. The purpose of the digital town library would be to generate insight and 
knowledge that can support understanding and decision making, and cycle knowledge and 
information back into the town square. This would be both a Gruberian collective knowledge 


system and a boundary infrastructure for matters vital to the future of humanity. 
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Cynthia Haynes & Jan Rune Holmevik 


Teleprompting Elekcriture 


“Writing is a physical effort... One runs the race with the horse, that is to say, with the 
thinking in its production. It is not an expressed, mathematical thinking, it’s a trail of images. 
And after all, writing is only the scribe who comes after, and who has an interest in going as 


fast as possible.” 


Héléne Cixous 


It is 1994. You see a command-line interface. A c> prompt invites you to log in to this essay’s 
directory. It is now 2013. A prompt indicates your Google glasses are ready to receive input. 
What a difference 20 years makes? Not so much. The directory for this collection of essays is 
accessed through the CyberText Yearbook Database, but the thought contained therein is not 
unlike what will have been (in the Nietzschean mode of the future perfect) a scrolling text 
readable on devices like virtual reality headsets, the progenitor of today’s Google glasses. 
Such devices are not so much an innovation in reading as a reading of innovation. Similarly, 
this collection is not so much a curated set of texts (or the preservation of conservative 
reading protocols) as they are texts that insist on a proto-curation: typo(-il)logically 
prototypical. We could use a more simple framework and just announce a redux of High 
Wired: On the Design, Use, and Theory of Educational MOOs (1998). The prompts for 
reading this directory of our collective redux are Movement (Haynes and Holmevik), Justice 
(Vitanza), Grammar (Butts), Web (Kuhn), Trauma (Sirc), and Reason (Ulmer). Or, if you 
prefer, we can regress even further and sit in the wings of an Elizabethan theatre and serve as 
prompters (book-holders) cueing the actors in this six-act play. Perhaps it will be kinder on 


our readers to set up a virtual teleprompter that gets things moving. 


a” 


Cynthia whispers: “Cue ‘Teleprompting Elekcriture’ 


The teleprompter has become as ubiquitous in politics as it has in entertainment, creating an 
historical convergence of reading protocols that depend on machine and movement. 
Teleprompted discourse is especially critical for politicians who must simulate their oratory 
skills, and who need to appeal/appear as if they are simultaneously informal and improvising. 


Such ethos is emblematic of Plato’s concern that writing would merely equip us with the 
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‘semblance’ of truth; “Once a thing is put in writing, it rolls about all over the place” 
(Phaedrus). So, too, the 24-hour news cycle (by some accounts less journalism than 
entertainment) situates the teleprompter both in front of the individual who ‘reads’ to viewers 
from a vertical syntagmatic streaming text, then reversed toward viewers and placed along 
the bottom of the screen in a horizontal paradigmatic text scroll that anticipates the next 
‘story’ or recaps previous stories. 


FOX News ticker 


There is something primitive (intuitive) about the way words appear!?, Conversely, there is 
something frightening (exhausting) about the way they dis/appear—scrolling upward with 
alarming speed, with the momentum of history, at the behest of time. In between, we inhabit 
the scroll bars, the space where movement and moment embrace. We witness language in 
action, in the languid flow of thought, the lurch of long-winded fragments, and the staccato 
bursts of out/landish play. We bid farewell to words with each keystroke, watching as they 
dwindle and fade from view. Imbuing them with invisible protection, we whisper, “may the 


force be with you.” We imagine them on their way—they travel as image. 
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Spiecde t 
Tre Phantom Menace 


Turmoil hae engulfed the Geltactia RepueWe . 
Tae taxation of trade routen Yo ovllsiwe eres 
systems is in dispute. . 


Moping te resocive the matter with 4& 
Slockade af deadiy battieships, the greety 
Trade Federation has stopped ali saipypng 

fo the small planet of Naboo. 


While the Congress of the Bepudl« 


eadlessiy debates this alarming  cnair 
Of events, the. Supreme Chancel\int Was 
secretly dispatched two Jedi Knignts, Wwe 
Glardians of peace and justice in the gaia, 


fo sett/e the conflict... 


Star Wars: Episode 1, The Phantom Menace© opening text crawl 


Who can forget the opening scene of Star Wars, the text marching into the infinite universe of 
the Galactic Republic. This filmic device tapped into our cultural experiences of moveable 


type, such as ticker-tape, cinema marquees, follow the bouncing ball sing- alongs, and 


vintage newsreel footage!*. It joined forces with a simple premise—moving text transforms 
thought into image and image into memory. It is perhaps uncharacteristic to claim that 
moving words stay with us longer. But we are interested in the un-character that un-does 
static print—that imagines us caught in a thicket of the thickest thieves: language and motion. 
There is, however, a crucial caveat, or noise, in this system: the material action of 
writing sets language into motion, whether by programming or raw physicality. Composition 
happens, to riff on Geoffrey Sirc and Jacques Derrida. And, as it happens, language speaks us 
and re-members us at the same time (in the same moment). By some accounts, a focus on 
writing and motion must start by studying the parts of writing we see, such as letters, words, 
i.e. printed static texts. John Trimbur argues that “studying and teaching typography as the 
culturally salient means of producing writing can help locate composers in the labor process 
and thereby contribute to the larger post-process work of rematerializing literacy” (192). As 
“the turn-of-the-century Austrian architect and graphic designer Alfred Loos put it so 
concisely, ‘One cannot speak a capital letter’” (191; qtd in Helfand 50). But Trimbur is 
narrowly focused on the typographical conventions that “[enable] us to see writing in 
material terms as letter-forms, printed pages, posters, computer screens” (192), while we are 
adjusting the focus to capture the images of writing in motion and the momentum that 
accrues in the backwash of memory. Through the many years we worked in MOOs, we came 
to understand such synchronous virtual space as a primary location of writing as images in 
motion. In other words, the appearance and disappearance of language inside a screen, the 


limits of which were beyond our vision, turned the scrollbar into a memory pole where words 
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unfurl in the prevailing and transient winds of writing’s warp-speed momentum. Typography 
became biography—the life-world of writing on the fly. 


Though the following exchange occurred in real time on October 9, 1999, it gives 
readers a sense of what we mean by ‘writing on the fly.’ William Gibson (author of the novel 
Neuromancer) logged in to Lingua MOO as part of a trAce Writing Community event in the 
U.K. We only had 30 minutes notice that he was logging in, so we hastily put out the word to 
Lingua users. He conversed with players in the MOO and created a ‘battered suitcase’ object 
into which you could place whatever MOO object you wanted. This is an excerpt of the 
MOO log that day: 


Helen says, "Bill's here" snapdragon waves at Bill Gibson. Jan waves at Bill_ Gibson. 
Bill_ Gibson says, "Hello, this really is Wm. Gibson, tho you won't believe me..."" 
Cynthia [to Bill_ Gibson]: We're honored to have you here at Lingua MOO! 

Tzen nods. 

traci says, "we're likely to believe just about anything" You laugh at traci. 


Mark Cole says, "Hi Bill. Enjoyed the talk downstairs. Any advice for a budding writer of 


speculative fiction (don't u hate labels?)" 


Bill_ Gibson says, "Thanks. This is the very last gig on my lightning UK All Tommorrow's 


Parties tour. 


Helen says, "How would a beginner get that ball of elastic bands going? (Bill's metaphor for 


writing a novel)" 
Helen says, "Anyone want me to buy them a signed book?" Tzen says, "Which book 1s it?" 
Nolan . 0 O ( and pay for it? whooohooo. ) 


Bill_ Gibson says, "Heinlein's advice: write, finish what you write, submit it, submit again 


when it's rejected." 

Jan smiles. 

Helen says, "All Tomorrow's Worlds" You take Neuromancer. 

Mark Cole says, "Thanks... have a jelly bean" You hand Neuromancer to Bill_ Gibson. 
Helen says, "Good advice Bill ;-)" Tzen says, "ah." 


Cynthia [to Bill Gibson]: yes, would you virtually sign my virtual copy of your book? :) 
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William Gibson interacting with Lingua MOO users (Oct 9, 1999) 


The MOO, as locus and instrument of linguistic register and re-collection, circum/scribes this 
composite image of writing and memory. Bruce Gronbeck reminds us that Aristotle makes a 
clear distinction between memory and recollection and tallies the attributes of recollection in 
his treatise De Memoria, “Recalling is always a matter of reconstructing ‘movement’ or 
sequences of action” (140; McKeon 451b-453a). For Aristotle, memory stems from 
recollection as such: “For remembering [which is the condicio sine qua non of recollecting] 
is the existence, potentially, in the mind of a movement capable of stimulating it to the 
desired movement, and this, as has been said, in such a way that the person should be moved 
[prompted to recollection] from within himself, i.e. in consequence of movements wholly 
contained within himself’ (McKeon 452a). 

Thus, early on our knowledge of how memory works is derived from Aristotle’s notion 
of motion contained. In her essay, “Habit as Memory Incarnate,” Marion Joan Francoz 
explains the containment model, the hydraulic model, and the physiological models of 
memory, advocating the latter and its association with habit. According to Francoz, “‘Image 
schemata,’ which Lakoff and Johnson propose as dynamic alternatives to abstract schematic 
representations in memory, find their most basic manifestation in the spatial aspect of the 


body, ‘from our experience of physical containment’ (Johnson, Body 21)” (14). 


But the movement we have in mind must also be a movement that is enduring, that 


ys 


gains momentum from the start, that keeps going. Viewed in this way, writing becomes a 
force, as Cixous writes, with which we contend and by which we leave our own trail of 
images. The trajectory of this essay follows three moments, or movements, along the trail of 
images we have left like bread crumbs for ‘the scribe that follows after’ and has somehow re- 


forged the relation between writing as image and learning via text in motion. 


*e* Connected ** 
Cynthia's Office 
Comfortable office where Cyndia meets friends and works on various projects. On the walls she has 
various Southwestern paintings and objects. Scattered among the books and bookcases are various 
plants like cactus and lamps for mood lighting. Cyndia Likes to work surrounded by things that re 
flecther taste and interests. 
Obvious exits: down to House of Blues 
You see Macintosh, T¥, VCR, Tapel, Cam, Paloma the Cat, and Yellow Rose here. 
Last connected Mon May 9 12:33:46 2011 CDT from 72.147.59.25 
There's a new edition of the newspaper. Type 'news' to see it. 
@go MediaN0o TY Studio 
MediaMO0 TY Studio 
WELCOME to the MMTY Studio. If there's any equipment you need just ‘activate 
Robbie' and ask him for it! 


SLIDE PROJECTORS 


| 
| T¥s VCRs TAPES CAMERAS 
| 
| CONVERSATIONAL BOTS 


Tape Editing Room W << MMTY STUDIO >> 
| ee ee \O/ 
| Tape Library | () ROBBIE 
Denar ee ce ee a aes ee eee Sy 4, een eee | 


Obvious exits: west to Tape Editing Room and N to West Media Ave. 

You see Broadcaster, VAX 11/788, Robbie, broadcast monitor, Tape Library, MediaMO0 TY Helicopter, 
drugz, 18@thMOOniversary, and ty here. 

Jan is here. 

Robbie says, "[to Cyn] I like mediaMO0 a lot, its fun to explore and learn new things." 


MediaMOO MMTV Studio (May 9, 2011;17th anniversary of our meeting on May 9, 1994) 


In 1994, when we first met in the text-based virtual community, MediaMOO, we quickly 
understood the power of writing in motion. The MOO is a blend of text and image, and of 
orality and literacy. Oral insofar as the interaction among writer/speakers in the MOO 
reproduces oral conversation via written text, literate insofar as the writing requires fluency 
to produce meaning. The interesting, and innovative, aspect of this phenomenon is that in the 
MOO tightening (and blurring) the orality/literacy split is achieved visually. Within months 
we created our own community using the LambdaMOO database, and within two years of 
creating Lingua MOO we had published our collection of essays, High Wired (University of 
Michigan Press), following which we created a graphical web-based interface called enCore 
Xpress, and soon thereafter, the 2nd edition of High Wired. Our task in the introduction to 
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High Wired was, we believed, to articulate (insofar as we could) a new name for such writing. 
We coined the term élekcriture, borrowing from the Greek for the beaming sun (Elektra) and 
French feminism’s notion of writing (/’ecriture feminine), to describe a thematic conjunction 
between electricity and the streams of writing that spill forth in a discourse that resists 


traditional ways of organizing and controlling the flow of conversation. 


And even after we combined the textual and graphical registers of meaning- production 
with a graphical interface that split the text side and the graphical side, élekcriture still 
dominated the production of meaning. Rhetorically, the design allowed for style to enhance 
input and for an intertextual-graphical interface to border the space in which learning takes 
place, while the web-based interface also made many MOO functions easier to learn and 
execute. But the fact that graphical MOO interfaces such as enCore Xpress had helped move 
MOO technology along at a pace in concert with other web-based communication software in 
the late 90s is not central to the idea we are promoting of text as image; we considered it 
merely a bonus. 


about help, =—sxpress who my stuf options mail’, ~—sobjeets. program 


The Collaboratory 


joOtcan version 0.1.20 S$ 
opyright (C) 1999-2001 Sindre Serensen and e 
Jan Rune Holmevik. All Rights Reserved. 


wizard = qui 


loOtcan comes with ABSOLUTELY NO WARRANTY 

his is free software, and you are welcome to redistribute 
it under certain conditions. (See the file COPYING in the 
jsource-code distribution, or the GNU GENERAL PUBLIC LICENSE 
Version 2, June 1991, at 

inttp: //www.gnu.org/copyleft/gpl.html) 


{Trying to connect to ‘a 

129.110.23.73 on port 8888 om we a Bed 
*** Connected *** 
{The Collaboratory 


aaa 


lYou view The Collaboratory... > an ft pr 
Zliks 

Obvious exits: [out] to Courtyard, [GNA] to GNA Forum : / 

(LinguaMOo), [CFF] to C-FEST Forum, {NDT] to The hh KF \ f 


erditorium, [PLA] to Plato's Pub: The Republic Bar and 
rill, [200] to LinguaZoo Courtyard, [OUR] to Our Town, 
(MLQ] to MOOLoquium, [S] to Safespace, [FSH] to First Shift, 
[Arcade] to The Arcade, [WAX] to Fourth Wall Theatre 
‘complex, [lost] to Lost Property Office, [Theoria] to 


heoria, [Street] to The Street, [Blvd] to Boulevard des We collaborate here...definition: "In the mediatrix, you throw 
sseins, {in} to The Backdoor Project yourself to others. You create through others. The media 
ast connected Fri Oct 14 03:52:55 2005 CDT from see ol realizes painfully that she must sacrifice her beloved 
x1=-6-00-11-24-08-51-38.k136.webspeed.dk cogito, her cherished institutionalism, her age-old desire for total 


control to a communal process-in-the-making" (Mark Taylor and 
Esa Saarinen, _Imagologies_ 8 


You see: Links: 


= —— Bccaasorsiory Projects —— 


LinguaMOO graphical interface, enCore Xpress (2005) 


Nineteen years ago we got to know one another in language, in real-time. It was both a 
‘home’ we could share and one we built for others to enter and build as they saw fit. We were 
living/writing in a visible text. The question of writing became a manifestation of personal 


and professional discourses, the crossing of which became for us an invisible boundary—we 
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did not distinguish between the space of our belonging to one another and to our academic 
others. It is akin to Bruno Latour’s reminder that “in the eyes of our critics the ozone hole 
above our heads, the moral law in our hearts, the autonomous text, may each be of interest, 
but only separately. That a delicate shuttle should have woven together the heavens, industry, 


texts, souls and moral law -- this remains uncanny, unthinkable, unseemly” (5). 


The second moment is really a fast forward ten years when MOOs began to wane as the 
graduate students who created, administered, and populated them moved on to “real” lives 
and jobs, and we found other platforms where writing in motion served as our template for 


play and purpose: Neverwinter Nights, Diablo II, Second Life, and World of Warcraft. 


Yet, in citing our own experiences we are somewhat torn. On the one hand, we believe 
the durability of these texts in motion seals the sagacity of our argument (not to mention the 
reality of our lives, which is hardly virtual any longer, though we tend not to make that 
distinction). On the other hand, as rhetoricians we understand the need for a critical eye. 
Roland Barthes expressed it in this manner: “...my desire to write on Photography 
corresponded to a discomfort I had always suffered from: the uneasiness of being a subject 
torn between two languages, one expressive, the other critical; and at the heart of this critical 
language, between several discourses, those of sociology, of semiology, and of 
psychoanalysis...” (Camera 8). This is how we approach writing about writing in visible 
texts; like Barthes, we are both “Operator” and “Spectator” (9). “The Photograph belongs to 
that class of laminated objects whose two leaves cannot be separated without destroying them 
both: the windowpane and the landscape, and why not: Good and Evil, desire and its object: 
dualities we can conceive but not perceive” (6). 

Barthes is instructive in an additional sense—as purveyor of the line between forms of 
visibility. In the static (print or web) iteration of this history, we understand that we cannot 
de/pict the motion of text we are de/scribing here. Even a “still” image (i.e., screenshot) of 
some MOO tran/script does not do justice to the movement experienced as graphé/flux (the 
flux of moving writing). But we can work with the concept of the photo/graph as theorized by 
Barthes because he re-animates it in order to ponder our pandemic belief in the invisibility of 
its animation of us. “Whatever it grants to vision and whatever its manner, a photograph is 
always invisible: it is not it that we see” (6). “In this glum desert, suddenly a specific 
photograph reaches me; it animates me, and J animate it. So that is how I must name the 
attraction which makes it exist: an animation. The photograph itself is in no way animated (I 
do not believe in ‘lifelike’ photographs), but it animates me: this is what creates every 
adventure” (20). 

There is, then, something that wants animating, that reveals itself when time and 


motion call certain features of text into the unconcealedness of typorganisms—of writing on 
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the move. Barthes meets Martin Heidegger at this juncture, redefining the ‘origin of the work 
of art,’ following the workness until we can see it at work. What Heidegger saw in a pair of 
worn out peasant shoes, Barthes sees in the instruments of time and photography: “For me 
the noise of Time is not sad: I love bells, clocks, watches—and I recall that at first the 
photographic implements were related to techniques of cabinetmaking and the machinery of 
precision: cameras, in short, were clocks for seeing, and perhaps in me someone very old still 
hears in the photographic mechanism the living sound of the wood” (15). The third moment 
along the trail of images comes into view now. Are MOOs and World of Warcraft like clocks 


for seeing writing? What happens in the seeing of composition as it happens? 


It is ttme—time that moves into a new topos where momentum gathers itself unto itself, 
where (it turns out) moments are re-turned to time. Who are we to think we owned them in 
the first place? We are so bound up in our sense of sovereign subjectivity that we dare to 
preface topos with its own ‘u’—unbounded topos—utopia. But in so doing, we have 
managed to create every dystopia known to humanity. MOOs and WoW are, thankfully, no 
utopias; they are more along the lines of what Alok Nandi calls a fluxtopia. According to 
Nandi: “Virtu/RE/alities explore the gap between virtuals, ideals and realities. Fluxtopia can 
only be understood in the act of attempting to achieve the traject of any flow. But how do we 
achieve what we mean by it if we do not know what it is, except that IT is in constant 
mutation, flowing apart?” (np). Nandi exploits our collective delusion that we can capture the 
flow of media by setting up various fluxtopic passages designed to foreground both delusion 
and passage. MOOs and WoW are portals into this “fluxography”; or, as Geoff Sirc might call 
it, this “fluxus-inflected practice” (“Fluxjoke” 3). The key to understanding how momentum 
assists memory rests not on the rests, or pauses, we inject in writing and reading, rather in the 
in/visible border between delusion and passage, one that is (hopefully) not subject to 
Aristotelian or Platonic border patrols. In synchronous writing environments we are lulled, by 
the momentum of language, into no complacent region of learning, but an active 
accumulation of meaning we commonly think of as memory. The movement of language, its 
marching momentum, lulls us into thinking we are pushing things along, when it is more 
accurate to say we are being pulled into a remembering machine without being aware of it. 
The question is how does momentum and language do this. And here we issue a patch to our 
earlier thinking on this topic by adding a small “t” to élekcriture—télekcriture. To underscore 
how télekcriture accomplishes this lulling, we should sample the most basic qualities of flux: 
rhetoric, rhythm, and reciprocity. 

As a rhetorical machine, té/ekcriture mixes language, writers, and distance, then 
reconfigures them as sustained contextual real-time interactivity. But distance itself also 


figures within language. Barthes suggests, as have others over the years, that all language is 
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rhetorical, that is, it is highly figurative. There are countless ways we attempt to maintain the 
distinction between two dimensions of language, the literal and figurative; but in the end, 
language is all figurative (Semiotic 82-93). In short, Barthes argues, “the meta-rhetorical 
expressions which attest to this belief are countless. Aristotle sees in 

it a taste for alienation: one must “distance oneself from ordinary locutions we feel in 

this respect the same impressions as in the presence of strangers or foreigners: style is to be 
given a foreign air, for what comes from far away excites admiration’” (88). There is, then, in 
language itself a dimension of distance, a sense in which words travel across time and 
distance in order to ‘mean’ something in the here and now. Words exhibit the wear and tear of 
distance and time, and no amount of anti-rhetorical rhetoric can undermine this fact. But 
critics like Paul Virilio misdirect their fears at teletechnologies (like MOOs and WoW) in an 
effort to restore to language (and thus to ourselves) a degree of nearness and sovereignty that 
seems to have slipped away (when it was never ours to begin with). As Virilio argues, 

“Tb ]etween the subjective and objective it seems we have no room for the ‘trajective,’ that 
being of movement from here to there, from one to the other, without which we will never 
achieve a profound understanding of the various regimes of perception of the world that have 
succeeded each other throughout the ages” (24). In short, he laments the “loss of the 
traveller’s tale” (25), he longs for the “essence of the path, the journey” (23). 

Whereas Nandi’s fluxtopia situates the trajective within the work (1.e., the act) of 
writing, Virilio situates it in the achievement of writing—the having travelled along a path. 
This is precisely the tension at work in the difference between print and electronic texts, 
something we think Richard Lanham missed in The Electronic Word, but not something 
Michael Joyce missed. In attempting to articulate the pulse of Carolyn Guyer’s phrase 
“tensional momentum,” Joyce finds evidence of a missing rhythm—a rhythm not present, 
literally, in print texts. But he’s torn, too. “And yet I know, in the way someone watches 
water slip through sand, that words are being displaced by image in those places where we 
spend our time online; know as well that images, especially moving ones, have long had their 
own syntax of the preliminary and the inevitable” (314). 

Writing in visible texts, like sand and water, flows at a rhythmic (ragged or silken) 
pace. In the exchange of languaging beings typing along this tempo-trajectory, reciprocity 
arises. It is woven by the ‘delicate shuttle’ of an/other interaction—sustained contextual real- 
time reciprocal interactivity. Reciprocal interaction partakes of a fluidity of movement related 
to (and determined by) tides and time. The backward (re-) and forward (-pro) movement of 
the tides, the ebbing and flowing of Oceanus in Homer’s //iad, lends its sense of fluid and 
cyclic language to real-time reciprocity. It is constant, continuing without intermission, 


steadily present, the constancy of real-time. Writing resists slowing down; it has its own force 
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of forward movement. In digital environments such as MOOs and WoW, this momentum 
rushes ahead of us and we are merely the scribes following after, somewhat engulfed by/in 


visible texts and set in motion by our words—in their current—on their way. 
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Deena Larsen 


Access within VR: Opening the Magic Doors to All 


Within each new technology lurks hidden obstacles. There are financial barriers to 
overcome, for those who struggle to put food on the table can not purchase the equipment or 
spare the time. There are physical obstacles for people who must maneuver this world in 
ways that differ from the norm. A cry that has often been offered in these situations is that we 
are working within unique media that simply can not trans(fer)(form) for all situations. Don’t 
ask the painter to explain art to the blind? Don't ask a symphony to exalt to the deaf? Perhaps. 
The wilderness is a wild and dangerous place, where only the intrepid can (ad)venture. Yet 
there are mountain trails with ropes and braille signs designed to provide a taste of the 
wilderness to the blind or widened slopes to give access to quiet forests for wheelchair users. 
We need to take a few minutes to explore setting up best practices for access to VR. Let's 


discuss solutions! 


¢ Supporting all channels. There are a myriad of physical capabilities needed to fully 
enjoy VR (mobility, sight, sound). How can we provide multiple channels within the VR 
environment to engage those people who can not participate in all of these channels (e.g., 
provide a canned experience for those with mobility issues, provide a caption for those 
with hearing issues, provide a running explanation for those with sight issues)? 
Conveying the VR kernel. We are using VR to communicate experiences and ideas in 
ways that could not be accomplished in other media. How then, can we convey the 


essence of this experience to those who can not physically participate in VR? 


Access within VR: Opening the Magic Doors to All 

“Hey, there is a great new VR Piece you just have to check out!” your friend exclaims, But 
you don’t have the equipment, so you cannot access the piece, and the Way to Save the 
Universe and the Grandest Message of Them All just passes you by. Or you actually have the 
equipment and you put it on. But the moment you do, you get so dizzy and so ill that you 
have to stop and lie down. Or you cannot hear the sound. Or the sound hurts you. Or you can 
not see the images. Or you cannot manipulate the controls. And again, the world passes you 


by—and you are left on the sidelines without ever getting the message. 
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A protest that has often been offered in these situations is that we are working within 
unique media that simply cannot trans(fer)(form) for all situations. Don’t ask the painter to 
explain art to the blind? Don't ask a symphony to exalt to the deaf? Perhaps. The wilderness 
is a wild and dangerous place, where only the intrepid can (ad)venture. Yet there are 
mountain trails with ropes and braille signs designed to provide a taste of the wilderness to 
the blind or widened slopes to give access to quiet forests for wheelchair users. We need to 


take a few minutes to explore setting up best practices for access to XR. 


Another objection is that everyone is grappling with a different barrier, so how can we 
actually address everyone’s needs? The good news about accessibility design is that one 
measure often works for more than one handicap or purpose. Curb cuts, for example, are not 
only terrific for wheelchair users, but also for people with strollers, heavy dolly loads, etc. 

So rather than seeing accessibility as an individual problem, address the needs of all users. 

Accessibility should be a primary consideration in developing the XR software as well as the 

pieces created using that software. After all, our implicit goals are to help as many people as 

we can use our softwares and grok our messages. Simply being aware of the needs is the first 

step. At the beginning of a software or creative project, ask: 

¢ How can we support all users and developers within the software and hardware 
environment? You need a myriad of physical capabilities to fully enjoy XR (mobility, 
sight, sound). How can we provide multiple channels within the XR environment to 
engage those people who cannot participate in all of these channels (e.g., provide a canned 
experience for those with mobility issues, provide a caption for those with hearing issues, 
provide a running explanation for those with sight issues)? 

¢ How can we create inclusive pieces to convey the messages using the software and 
hardware? We are using XR to communicate experiences and ideas in ways that could 
not be accomplished in other media. How then, can we convey the essence of this 
experience to everyone, regardless of ability, age, race, gender, etc.? 

Luckily, you do not have to answer these questions by yourself! There is an entire community 

dedicated to accessibility and inclusive design within XR. There are best practices to 

minimize barriers and provide multiple channels to get the message, including: 


¢ Applying Universal Design principles to the technology 


¢ Creating more than one channel to convey the message (providing both audio and visual 
channels, text descriptions of actions, using both colors and shapes rather than color alone 
etc.) 


¢ Avoiding triggers (flashing lights, dizzying motions, etc.). If these can not be avoided, at 


least provide warnings 
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Following guidelines for accessibility, including the W3C working group (https:// 
www.w3.org/TR/xaur/) 

Working with XR developers and researchers (https://xraccess.org/symposium/ and 
https://xraccess.org/research/)Examples of tips for writing pieces include: 

Have adjustable volumes. Let the user determine the loudness. 

Have warnings just like we do with flashing lights for seizures (this artwork/music 
contains high pitches) 

Have captions (this helps hard of hearing, second language speakers, people on the 
spectrum, and others as well) 

Try to provide multiple channels for the same information. Use visual patterns along with 
music or text descriptions. If you cannot reproduce the whole thing, carve out one piece 
that can be reached. For example, with VR, try to explain or introduce one small piece so 


people can get a feeling for it. 


Use metadata consistently to describe pieces and their accessibility, including: 


Warnings (pitch, flashing lights, inconsistent volume levels--standardized warnings would 
be great) 

Content delivery (sound, imagery, structure, navigation--explain where the content is) 
Content duplication effort (yes/no--does the piece try to convey the spirit or the same 
meaning in different ways? So having captions would be yes, but having just sound music 
would be no.)This is of course, the tip of the iceberg. The key here is to open up these 


wonderful realms of possibility in XR to everyone and lower all barriers. 
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Dene Grigar & Richard Snyder 
Metadata for Access: VR and Beyond 


by Dene Grigar and Richard Snyder, Electronic Literature Lab, Washington State University 


Vancouver 


Abstract 


Interacting with virtual reality (VR) environments requires multiple sensory modalities 
associated with time, sight, sound, touch, haptic feedback, gesture, kinesthetic involvement, 
motion, proprioception, and interoception—yet metadata schemas used for repositories and 
databases do not offer controlled vocabularies that describe VR works to visitors. 

This essay outlines the controlled vocabularies devised for the Electronic Literature 
Organization’s museum/library The NEXT. Called ELMS (Extended eLectronic Metadata 
Schema), this framework makes it possible for physically disabled visitors and those with 
sensory sensitivities to know what kind of experience to expect from a VR work so that they 
can make informed decisions about how best to engage with it. In this way accessibility has 
been envisioned so that a// visitors are equally enabled to act upon their interest in accessing 
works collected at The NEXT. 


Introduction: Proof of Concept 

Turning their head slowly, the player spots five neon green pins in the horizon and aims their 
controller at the one peeking behind the conical dark-green cedar. The player is situated amid 
a strange, bright blue terrain undulating beneath a cloudy gray and blue sky. In the 
background they hear voices chattering and laughing softly. Moving their head further to the 
left, the player sees more green pins hovering over bleak squat buildings and an earth-like, 
blue globe. It seems like they are walking toward the globe, and as they get closer, they see a 
bookshelf sunk backwards into the ground. Approaching it, the chattering grows loud and 
then stops. 

This is one of the scenes in Everyone at this party is dead / Cardamom of the Dead by 
Caitlin Fisher, one of the first VR literary works produced for the Oculus Rift. Published in 
2014 in the Electronic Literature Organization’s Electronic Literature Collection 3 (ELC3), it 
is now hosted at The NEXT. 

Like the 3000 other works of born-digital art, literature, and games that The NEXT 


holds, Fisher’s VR narrative is presented in its own exhibition space. A carousel of still shots 
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from the work presents visitors with highlights from the work. The description of the work, 
cited from the ELC3, provides information about the storyline, the artist’s vision, and its 
production history. To the right is a sidebar containing the “Version Information”—metadata 
built on the MODS schema detailing bibliographic information expected from a scholarly 
database. This information includes the author’s name, date of publication, publisher, and 
language all associated with the 1.0 version of Fisher’s work. Visitors, however, also see 
additional information that goes beyond that provided by MODS: the work’s digital qualities, 
its genre, the sensory modalities evoked when experiencing the work, its accessibility, 
original media format, authoring platform, and peripheral dependencies. These are controlled 
vocabularies that move beyond the bibliographic and, instead, provide visitors with the 
information they need in order to experience the work. In this context, Everyone at this party 
is dead / Cardamom of the Dead alerts visitors that the work involves kinesthetic 
involvement, proprioception, sight, sound, graphical and spatial navigation, and that it was 


built with Unity and requires a VR headset. 


About The NEXT’s Extended Metadata Schema 

The metadata schema for The NEXT, ELMS or the “Extended eLectronic Metadata Schema,” 
is the framework developed to provide a common understanding of the highly complex, 
interactive, digital artifacts, like Fisher’s, held in its collections. 

Because The NEXT collects and hosts a wide variety of interactive media pertaining to 
digital art and writing—the bulk of which it makes freely available for access and download 
in their original formats or in formats that have been preserved through migration and 
emulation—its schema both utilizes and extends the Metadata Object Description Schema 
(MODS) maintained by the Network Development and MARC Standards Office of the 
Library of Congress. By extending MODS, The NEXT attends to the media specificity of the 
works, an approach to the analysis of digital objects suggested by theorist N. Katherine 
Hayles in Writing Machines (Hayles, 2002) and also reflected in taxonomies created by the 
global, scholarly federation, the Consortium on Electronic Literature (CELL), over a decade 
ago!>, 

At the heart of ELMS is the contention that visitors accessing a work at The NEXT 
need to be made aware of its hardware, software, peripheral specifications, and other salient 
features so that it can be experienced fully. Taxonomies developed for extending MODS 
include Software Dependency(ies), Authoring Platform(s), Hardware Dependency(ies), 
Peripheral Dependency(ies), Computer Language(s), Digital Quality(ies), Sensory 
Modality(ies), and Genre(s). 


Equally important, disabled visitors need to know the physical requirements of a work 
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in order to prepare for the experience via assistive technologies and/or other methods. Thus, 
ELMS’s metadata has been further extended to meet the needs of disabled visitors and those 
with sensory sensitivities so that they know the kind of experiences a work involves and can 


make informed decisions about engaging with it. Specifically, the system, aligned with crip 


theory and relaxed performance methodology'®, pairs a controlled vocabulary that extends 


traditional metadata fields to include those related to disability access—what we refer to as 


sensory modalities—with descriptive language expressed in Plain/Simple English!” that 


further details particular hazards disabled visitors need to know before encountering a work. 


Because the participatory, interactive, and experiential qualities of born-digital art, 
literature, and games involve what Vince Dziekan refers to as “virtuality” and a sense of 
“liveness” (Dziekan, 2012), principles underlying the development of the space and the 
treatment of the works it holds align well with practices associated with live performance. 
The concept of the performative nature of computers has been raised early on by scholars, 
such as Brenda Laurel and Janet Murray. Thus, in extending The NEXT’s metadata schema to 
address a multitude of disabilities and sensory sensitivities, ELMS’s approach to access 
draws upon the practice of relaxed performance visual story guides, similar to those created 
for relaxed theater/concert performances, etc., when creating a statement for each work in 
The NEXT. These statements outline in Plain Language what a visitor can expect from their 
experience with a work and are tied directly to controlled vocabularies in the metadata that 


make it searchable and able to be filtered for a customized experience. 


A relaxed performance offers a comfortable, welcoming visitor experience that 
accommodates a wide range of needs. Disabled people and those with sensory sensitivities 
are able to participate and enjoy an event as valued patrons (“Sensory Relaxed 
Performances’’). A common practice for relaxed performances is the distribution of a guide 
that lets visitors know in advance what to expect at the performance and how it has been 
modified to accommodate specific needs. In context of The NEXT, the metadata located in 
the sidebar of an individual work’s exhibition space describes its unique, searchable features. 
The section called “experiencing the work” that follows the description of a work’s content 
provides the kind of detailed information, written in plain and clear language, that conveys to 


the visitor what to expect from the work and when specific actions occur. 


Applying ELMS to VR Narratives 

Going back to Fisher’s VR narrative, visitors would be alerted to the fleeting text that appears 
briefly and then disappears. They need to know that text moves across the environment and 
that the reading time is also brief. If they have color-blindness associated with distinguishing 


greens and blues, tritanomaly for example, they may not be able to differentiate easily the 
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color of the pins and of other objects such as the cedar tree, many of which carry important 
information for navigating the experience. They should be also aware that much of the poetic 
content is communicated over audio, and that the sound oscillates between soft and loud and, 
so, could be challenging to sensitive visitors. They would need to know that it is necessary to 
manage a controller and vibrations occur to signal that the visitor has successfully targeted a 
green pin. Head movements are also required. Some of the work’s meaning is communicated 
spatially via perception of artificial depth. Finally, visitors need to be alerted that they may be 


affected with internal sensations, such as nausea or dizziness, due to the VR experience. 


Everyone At This Party Is Dead 


Information 


CREATOR'S) 
F 


1 COPY IN THE NEXT 


The NEXT ’s Exhibition Space for Caitlin Fisher s Everyone at this party is dead / Cardamom 
of the Dead with Controlled Vocabularies and Statement for Disabled Visitors and those with 


Sensory Sensitivities 
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Final Thoughts 
The ELMS metadata schema starts with the premise that all visitors to The NEXT need some 


type of accommodation to access the born-digital works held in its collections, whether it is 
information relating to the hardware a hypertext novel needs to function or the sensory 
modalities it evokes as it is experienced. Visitors who use screen readers, for example, should 
know in advance that they will need this technology to access a net art piece that requires 
sight; likewise, those who do not have access to an Oculus Rift headset will be informed 
when a work, like Fisher’s, requires one. In this way all visitors are equally enabled to act 


upon their interest in accessing works collected at The NEXT. 
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Eduardo Kac 


Space Art: My Trajectory 


This paper traces the author’s trajectory in space art. It starts in 1986, when he first conceived 
of a holographic poem to be sent in the direction of the Andromeda galaxy, and continues into 
the twenty-first century through several works, including Inner Telescope, realized with the 
cooperation of French astronaut Thomas Pesquet aboard the International Space Station (ISS) 
in 2017. The author discusses his theoretical and practical involvement with space-related 
materials and processes. Special attention is given to his space artwork Adsum, conceived for 
the Moon. 


Introduction 


I started my career in 1980, with a multimedia practice that integrated poetry, performance, 
and the visual arts. Beginning in 1982, I pivoted towards an engagement with technology as 
my creative medium, a sustained orientation that marks its fourth decade in 2022. Albeit 


lesser known than my other bodies of work, space art has been central to my interests since 


the early 1980s!®. In what follows I will revisit some of the key moments in my space art 


trajectory. 
Agora: a holopoem to be sent to Andromeda 


In 1983 I introduced a new art form that I named holographic poetry, or holopoetry (Kac, 
2007), which consisted in the use of unique properties of holography to create poems that 
floated in the air and changed their configurations according to the relative position of the 
observer. One of the fundamental tenets of holopoetry is what I called antigravitropism, 1.e., 
the use of language in a way that does not follow the perceivable effect of gravity on writing. 
In other words, the creation of works that, albeit produced on Earth, were not limited by the 
action of gravity on matter because the holopoems were composed of light (i.e., photons, 
massless particles). This meant that, contrary to telluric objects, the letters and words in the 
holopoems were anti-gravitropic; they hovered freely outside, inside, or through the surface 
of the recording medium (i.e., holographic film or glass plate). Through the manipulation of 


this plasticity I created shape-shifting works; I produced a word-image continuum that, from 
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the point of view of a moving observer, exists in a constant state of flux. I developed this art 


form until 1993, resulting in a body of work comprised of twenty-four pieces. 


In 1986 I created my first space artwork, a holopoem to be sent in the direction of the 
Andromeda galaxy (see Kac 1). This work is in Portuguese and is entitled Agora (agora, in 
English). In the work itself, we see the word Agora (now, in English) rendered in wireframe. 
The difference between the two words, in Portuguese, is the acute accent, used to mark the 
vowel height. With this diacritic mark, the word makes reference to space; without, it makes 
reference to time. Taken together, they allude to the intertwined relationship between space 


and time. 


(Kac 1) Agora, holopoem conceived to be sent in the direction of the Andromeda galaxy (not 
launched). Kac, 1986. 


As seen in the holopoem, the letters of AGORA (all in uppercase, in order to create a weight 
equivalence between the letters) are written three-dimensionally with a wireframe font. This 
enabled all strokes and angles of the letters to be seen simultaneously, dramatizing their 
immaterial form through emphasis on the outlines. Thus, the ‘emptiness’ of the letters echoes 
the perceived ‘void’ of space. 

Agora was conceived to be released in space and propelled in the direction of the 


Andromeda galaxy, like a message in a bottle travelling through the vacuum of space. Agora 
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was made with an angle of incidence of 45 degrees. This means that whenever light would 
shine on the hologram at approximately 45 degrees, the hologram would ‘irradiate’ the word 
AGORA in wireframe, visible to the naked eye. My vision for this work was that, throughout 
its trajectory in space, it would function as an ‘intermittent star’: whenever light would strike 
it at approximately 45 degrees, it would diffract the incoming light and output a wavefront 
that would be visible as the word AGORA. As it tumbled amid the darkness of the cosmos, it 
would occasionally ‘emit’ light in different directions, always encoded with the urgency of its 


message: ‘now’. 


Spacescapes 


In 1989, I transmitted from Chicago my artwork Spacescapes (Kac, 2022) via Slow-Scan 
Television (SSTV) simultaneously to Pittsburgh (to the DAX Group) and to Boston (to a 
local group of artists). The transmission took place in the context of the Three-City Link 
event, a three-node ephemeral network configured specifically for the event. 
SSTV was an early type of videophone that allowed the transmission/reception of sequential 
still video images over regular phone lines. On average, it took from eight to twelve seconds 
to transmit each image. 

In Spacescapes (see Kac. 2), an alternating sequence of satellite views and microscopic 
images of digital circuits fused into one another at the receiver's end, forming an electronic 


palimpsest in which large and small merged. 
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(Kac 2) Spacescapes, slow-scan television, screen, telephone line, satellite and microchip 


images. Example of a transitional frame as seen by recipients. Kac, 1989. 


This work explores the analogy between patterns seen up close at a minute scale and forms 
revealed at great distances. Spacescapes creatively manipulates an intrinsic characteristic of 
the system, which was to scan, from top to bottom, the incoming image over the preceding 
one. As a result, their amalgamation took place at the receiver’s end, producing a continuous 
transformation of landscapes seen top-down—in which it was very difficult to discern what 
was the Earth seen from a satellite and what was a microchip seen through a microscope. 
Through this work I wanted to convey an aesthetic of magnitudes, alternating 
perspectives from the inward motion into a microscope to the vantage point above the surface 
of the Earth, and back again, continuously. The transitions between the two deliver a one-of- 
a-kind experience, interlaced as they are with the same electronic glow. Ultimately, the 
uninterrupted fusion of ultra-close and ultra-far images suggests the interconnectedness of the 


infinitesimal and the monumental, and the awe of our relative position in the world. 


Monogram 


My ink drawing Monogram (Kac, 2022), which evokes an orbital trajectory, a rising rocket, 
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and a moon (and is also my emblematic signature), flew to Saturn on the Cassini spacecraft in 
1997. Traditionally, a signature is a complement to an artwork, a graphic surplus often placed 
on the lower right corner of a picture or at the bottom of an object, to indicate authorship and 
authenticity. However, in the case of Monogram, I elevate the signature to the condition of 
artwork itself by drawing attention to its visual qualities and semantic resonances. The 
curlicues of Monogram configure stylized representations of visual elements unique to space 
exploration (see Kac. 3). Its iterability assures its legibility in the absence of the sender or a 


specific addressee. 


(Kac 3) Monogram, Kac's ink drawing, which evokes an orbital trajectory, a rising rocket 
and a moon (and is also the artist's emblematic signature), flew to Saturn on the Cassini 
spacecraft in 1997. Cassini entered orbit around Saturn in 2004. Kac, 1996. 


The original, wavy ink drawing was digitized and included in a DVD, which was placed 
between two pieces of aluminum to protect it from micrometeoroid impacts, and mounted to 
the side of the two-story-tall Cassini spacecraft beneath a pallet carrying cameras and other 
space instruments that were used to study the Saturnian system. A patch of thermal blanket 
material was installed over the disk package. 

The Titan [VB/Centaur rocket carried the Cassini spacecraft, as they launched from 
Cape Canaveral Air Force Station's Launch Complex 40, on October 15, 1997. Cassini 
entered orbit around the giant planet in 2004 and completed 294 Saturn orbits. On September 
15, 2017, Cassini deliberately dove into Saturn's atmosphere, burning up and disintegrating, 
in order to prevent the contamination of Saturnian moons targeted for research on the 
possibility of life. 


This means that the artwork, with each curve sweeping into another, was in deep space 
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for twenty years, a meaningful fact in itself and also for its symbolism: the presence in the 
cosmos of a unique physical mark that stands for the individual maker, a personal glyph, a 
manu propria sign that points to the signer and voluntarily expresses it. A signature is 
indexical by definition, that is, it is a signifier that is physically connected to the signified, it 
unequivocally affirms the existence (in the present or the past) of the signee by contiguity. A 
“signature work” means an emblematic piece, one that epitomizes the aesthetic vision of the 
artist. The loops and curves of Monogram define, instead, a work-cum-signature, a consistent 


graphic pattern made of variable twirling traces that, overall, can be repeated. 


If today we already travel telerobotically between the planets of the Solar System (with 
the exception of Voyager, which has flown beyond the heliopause and has entered interstellar 
space), in the future crewed interplanetary spaceflight will become more common. In this 
new context, art will be a meaningful participant in the journey. In its singular, swift lines, 


Monogram seeks to express the vitality of cultural practice in interplanetary space. 


The Lepus Constellation Suite 


Created, produced and transmitted in 2009 from Cape Canaveral, Florida to the Lepus 
Constellation, the suite is composed of five line drawings that were also rendered as five 
engraved and painted steel discs, measuring 20 inches in diameter each (Kac, 2022) [5]. 

The Lepus Constellation Suite is part of a larger series entitled Lagoglyphs, ongoing 
since 2006, in which I develop a leporimorph or rabbitographic form of writing. The larger 
series includes prints, murals, sculptures, paintings, an algorithmic animation, and satellite 
works created specifically for visualization in Google Earth (more on the latter below). As 
visual language that alludes to meaning but resists interpretation, the Lagoglyphs series 
stands as the counterpoint to the barrage of discourses generated through, with, and around 
my GFP Bunny (a green-glowing transgenic bunny, called Alba, that I created in 2000, and 
that has been featured in exhibitions and publications worldwide). 

The pictograms that make up the Lagoglyphs are visual symbols representing Alba 
rather than the sounds or phonemes of words. Devoid of characters and phonetic symbols, 
devoid of syllabic and logographic meaning, the Lagoglyphs function through a repertoire of 
gestures, textures, forms, juxtapositions, superpositions, opacities, transparencies, and 
ligatures. These coalesce into an idioglossic and polyvalent script structured through visual 
compositional units that multiply rather than circumscribe meanings. 

Composed of double-mark calligraphic units (one in green, the other in black), 


the Lagoglyphs evoke the birth of writing (as in cuneiform script, hieroglyphic orthography, 
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or ideography). However, they deliberately oscillate between monoreferentiality (always 
Alba) and the patterns of a visual idiolect (my own). In so doing, the Lagoglyphs ultimately 
form a kind of pictorial idioglossia or cryptolanguage. 

In the specific case of The Lepus Constellation Suite, the five lagoglyphic messages 
were transmitted towards the Lepus Constellation (below Orion) on March 13, 2009, from 
Cape Canaveral, Florida (see Kac. 4). The transmission was carried out by Deep Space 
Communications Network, a private organization near the Kennedy Space Center. At a 
frequency of 6105 MHz, the transmission was accomplished through high-powered klystron 
amplifiers connected by a traveling wave-guide to a five-meter parabolic dish antenna. Based 
upon its stellar characteristics and distance from Earth, Gamma Leporis (a star in the Lepus 
constellation that is approximately 29 light-years from Earth) is considered a high-priority 
target for NASA's Terrestrial Planet Finder mission. The Lepus Constellation Suite will arrive 


in its vicinity in 2038. 


(Kac 4) The Lepus Constellation Suite, 5 engraved and painted steel discs (20 inches 
diameter each) with lagoglyphic interstellar messages transmitted to the Lepus Constellation 
on March 13, 2009 from Cape Canaveral, Florida. Illustrated is disc #3. Kac, 2009. 


Lagoogleglyphs 


Another suite of works in the Lagoglyphs series is entitled Lagoogleglyphs (2009-ongoing) 
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(Kac, 2022) [6], space artworks that inscribe pixelated lagoglyphs (my abovementioned green 
rabbit glyphs) onto the environment and make them visible to the world through the 
perspective of satellites. These pixelated artworks are created at a global scale and can be 
experienced in person at their respective venues, directly via satellites, or through Google's 
geographic search engine (hence their name). In the latter case, the viewer may choose to see 


the work in one of the following three options: 

1) the familiar Google Maps (in satellite view), 

2) Google Earth (which can be accessed by typing “Google Earth” on a web browser) or 
3) the equally free Google Earth Pro app (which has the additional feature of allowing the 


viewer to see a map over time by activating the Historical Imagery slider). 


In addition to the distributed artworks (seen in person; online; from space), I have created a 
video for each individual Lagoogleglyph by capturing, in Google Earth Pro, the view from 
space all the way down to the eye of the rabbit glyph on Earth (and back again to outer 
space). The videos loop, are silent, and average one minute in duration. Between 2009 and 
2022, I have created five Lagoogleglyphs (and their respective videos) in the following 
locations: 1) Rio de Janeiro; 2) Mallorca; 3) London (see Kac. 5); 4) Strasbourg; and 5) 
Geneva. The videos #1 through #4 were exhibited together, for the first time, at the Venice 
Biennale, from April 20 to November 27, 2022. 


Lagoogleglyph 3, space artwork realized in London to be seen by satellites, to be experienced 
in person and/or through Google Maps (satellite view), Google Earth or the Google Earth 
Pro app. It measures 20 x 15m (65.6 x 49.2 ft). Kac, 2018. 
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Lagoogleglyph 1 was implemented on the roof of the art center Oi Futuro, in Rio de Janeiro, 
in 2009, as part of my solo exhibition Lagoglyphs, Biotopes and Transgenic Works, curated 
by Christiane Paul, on view at Oi Futuro from January 25th to March 30th, 2010. Printed on a 
large, polygonal canvas measuring approximately 8 x 17 meters, it covered the entire roof of 
the building. For the inaugural work in the series, I custom-ordered a WorldView-2 satellite 
photograph, which was subsequently incorporated by Google into its search engine by pulling 
it from the DigitalGlobe catalogue. Even though the roof installation was ephemeral, the 
work still remains visible in Google Earth Pro. To see it, the reader is encouraged to drag the 
Google Earth Pro time slider to the date of January 2010. The time slider is accessible 
through a topbar icon that consists of a clock capped by an arrow pointing counterclockwise. 
The original Lagoogleglyph | canvas, together with documentation material, is in the 
permanent collection of the Museu de Arte do Rio-MAR, Rio de Janeiro. 

Lagoogleglyph 2 was also printed on canvas. This time, the work measured approximately 10 
x 12 m (32 x 34 ft) and was displayed on the roof of Es Baluard Museum of Modern and 
Contemporary Art, Palma de Mallorca, Spain, in 2015. The work was commissioned by the 
museum and is also in its permanent collection. Its image was captured by the WorldView-3 
satellite. 

Lagoogleglyph 3 and Lagoogleglyph 4 were both made and exhibited in 2018; the 
former in London and the latter in Strasbourg. This time, instead of rooftops, both works 
were installed on the ground and were composed of grass and field marking paint. In addition 
to their distinct compositions, they also differ in scale and execution. Lagoogleglyph 3 
measured 20 x 15 m (65.6 x 49.2 ft). It was painted directly on the grass at Finsbury Park, 
London, on the occasion of my solo exhibition Poetry for Animals, Machines and Aliens: The 
Art of Eduardo Kac, realized at Furtherfield, an art center located at Finsbury Park, from 
April 7th to May 28th 2018, and curated by Andrew Prescott and Bronac Ferran. 
Lagoogleglyph 4 measured approximately 8.5 x 4.2 m (28 x 14 ft). It was made of sod 
squares and installed in the garden of the art center Apollonia — European Art Exchanges, in 
Strasbourg. 

Lagoogleglyph 5 was installed in the Cimetiére de Plainpalais, generally known as 
Cimetiére des Rois, in Geneva, in the context of the group exhibition Open End 2, from 
September 15 to January 31, 2022, organized by Vincent Du Bois. The Cimetiére des Rois 
(Cemetery of Kings) is renowned for being the final resting place of notables such as Jorge 
Luis Borges and Jean Piaget, and for hosting group shows with artists such as Sophie Calle 
and Olafur Eliasson. 
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Inner Telescope 


After ten years of work as artist-in-residence at the Observatoire de l'Espace (Space 
Observatory), the cultural lab of the French Space Agency (CNES), in 2017 my artwork Inner 
Telescope was realized on the International Space Station (ISS) with the cooperation of 
French astronaut Thomas Pesquet (see Kac. 6). Inner Telescope was specifically conceived 
for zero gravity and was not brought from Earth: it was made in space by Pesquet following 
my instructions. The fact that Inner Telescope was made in space is symbolically significant 
because humans will spend ever more time outside the Earth and, thus, will originate a 
genuine new culture in space. Art will play an important role in this new cultural phase. As 
the first artwork specifically conceived for zero gravity to be literally made in space, Inner 
Telescope opens the way for a sustained art-making activity beyond our terrestrial dwelling. 
Inner Telescope was made from materials already available in the space station. It 
consists of a form that has neither top nor bottom, neither front nor back. Viewed from a 
certain angle, it reveals the French word “MOI [meaning “me”, or "myself"]; from another 
point of view one sees a human figure with its umbilical cord cut. This “MOI“ stands for the 
collective self, evoking humanity, and the cut umbilical cord represents our liberation from 
gravitational limits. Inner Telescope is an instrument of observation and poetic reflection, 
which leads us to rethink our relationship with the world and our position in the Universe. 
In the course of developing the work, I created a protocol for its fabrication aboard the 
ISS, which I personally transmitted to Pesquet in 2016 during our work session at ESA’s 


European Astronaut Centre, a training facility in Cologne. 
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(Kac 6) Inner Telescope in the cupola, ISS. Kac, 2017. 


In addition, I also created a separate protocol for the video documentation of the work aboard 
the ISS. From the raw footage produced by Pesquet I edited a 12-min video, which is an 
artwork in itself; in it we see Inner Telescope being made in the Columbus module, its 
perambulation through the station, away from the module and in the direction of the cupola, 
and finally its arrival at the cupola with the Earth in the background. I published this video in 
a limited edition of five copies. The video Télescope intérieur (Inner Telescope) is in the 
permanent collection of Les Abattoirs, Museum - Frac Occitanie Toulouse, a public 
institution that houses both a French museum and the Regional Fund for Contemporary Art. I 
have made additional artworks in the Inner Telescope series, including drawings, 
photographs, prints, embroideries, installations, and artist’s books. 

The project also included the documentary film "Inner Telescope, a Space Artwork by 
Eduardo Kac", directed by Virgile Novarina (French, with English subtitles, 2017). Since its 
release, the documentary has been continuously screened internationally at museums, theaters 


and other places, including notable venues such as the Louvre Museum, Paris. The film was 


published as a DVD in 2017!”. In addition, the bilingual book Eduardo Kac: Télescope 
intérieur / Inner Telescope was edited by Gérard Azoulay and published by the Observatoire 
de L'Espace/CNES, Paris, in 2021 (Kac, 2021). 

My Space Poetry manifesto was published in 2007 (Kac, 2007), when I started to work 
on Inner Telescope. In 2017, I finally realized the dream of challenging the limits of gravity I 
had pursued for more than thirty years: the creation, production, and experience of a work 
directly in outer space. The astronaut's mission was entitled "Proxima" and was coordinated 
by the European Space Agency (ESA). Inner Telescope was coordinated by L'Observatoire de 
l'Espace, the cultural lab of the French Space Agency. 


Adsum, an artwork for the Moon 


Conceived for the Moon, Adsum is a cubic glass sculpture inside of which four symbols are 
laser engraved (see Kac. 7). The cube measures 1x1x1cm (0.4x0.4x0.4”). The symbols are 
positioned one in front of the other, thus forming a spatial poem inside the solid glass cube 
that can be read in any direction (Kac, 2022). ‘Adsum’ means ‘Here I am’ in Latin, as used to 


indicate that the speaker is present (equivalent to the exclamation ‘here!’ in a roll call). 
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5 “Wes 


Adsum (in progress), space artwork (laser-etched optical glass), IxlxIcm (0.4x0.4x0.4"). 
Kac, 2022. 


To create this space poem, I developed a new typeface in which the letter ““N” takes the form 
of an hourglass and the letter “S” has the shape of the infinity symbol. This makes the work 
legible from any point of view within the cube. The two other letters, which stand between 
“N” and “S,” are a lowercase “o” and an uppercase “O” (evoking the Moon and the Earth, 
respectively). Taken together, it is always possible to read either “NoOS” or “SOoN” in three 
dimensions. 

In addition, the design and spatial arrangement of the letters also produce a purely visual 
experience: a reversible transition from hourglass (representing human experience of time) to 
infinity (representing cosmic time). The shift in scale from the lowercase 'o' to the uppercase 


'O' suggests a zoom effect going from time as apprehended by human cognition to the 
temporal expanse of the universe (and vice-versa)””. 


Adsum flew on an Antares 230+ rocket from Wallops Flight Facility, Virginia, to the 
International Space Station on February 19, 2022. The artwork was aboard Cygnus NG-17 
(Northrop Grumman-17), a cargo resupply mission of the Northrop Grumman Cygnus 


spacecraft to the ISS under the Commercial Resupply Services (CRS) contract with NASA?!, 
Adsum was housed in the Columbus module of the ISS. 

Adsum’s journey to the ISS in 2022, traversing anaerobic, radioactive coldness, was a 
test to confirm its readiness for space flight. Adsum will progressively approach the Moon in 


three additional steps, each with its own visual and material version: 1) Adsum (regex 
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version) is composed of typographic characters and is designed to orbit our nearest celestial 


neighbor, in digital form, on a USB drive aboard the Orion spacecraft’; 2) Adsum (planar 
version) will arrive on the Moon aboard Intuitive Machines’ Nova-C lander, etched on a 
Galactic Legacy Labs’ nickel nanofiche disk; 3) Finally, Adsum (lander version), identical to 
the cubic glass sculpture that flew to the ISS, will be aboard an Astrobotic lander that will 
arrive on the Moon NET 2023. As a result, both the planar and the sculptural versions of 
Adsum will literally be on the Moon, there staying for endless time, protected from the harsh 
lunar environment inside their respective landers, awaiting discovery by future space 
explorers—possibly inhabitants of the first lunar settlements. 

In order to communicate the work’s message on Earth, I have created a series of pieces 
that can be exhibited together or separately, including a limited edition of the laser-engraved 
glass cube itself, dozens of ink drawings, and a looping video in which we see the minute 
cube up close, continuously turning to reveal its multiple meanings, with the myriad 
reflections and refractions of the symbols adding a unique aesthetic quality to the experience. 
Adsum embodies and expresses the fugacity of the human condition and our awe before the 


cosmos. 
Conclusion 


As demonstrated in the preceding pages, since the 1980s I have been theorizing and 
producing art and poetry that challenge the limits of gravity. It is my conviction that space art 
can be pursued in many different ways, all equally valid in their respective approaches. 
However, in light of the fact that what enables space exploration is its underpinning material 
reality, it is clear that art that directly engages with the technologies of space possesses a 
particularly distinct characteristic. Not in the sense of style or form, but in the sense of its 
contiguity with human presence and agency outside of our home planet. Making art on Earth 
through the use of space media (such as satellites), making art directly in space (in Earth’s 
orbit or beyond), or making art on Earth specifically to be flown to space — all are modes of 
creation and production that correspondingly have the symbolic and factual meaning of 
pointing to a future in which art and space exploration are intrinsically, and routinely, 
intertwined. Ultimately, art that directly engages with the technologies of space has the 
potential to contribute to the creation and development of what we may call “space native” 


culture—one created in space and for space. 
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Why PDF is the wrong format to bring text to XR and why the Web 
with proper provenance and responsive design from stylesheets is all 


we need 


For the Future of Text numerous discussions started on the premise that PDF is an interesting 
format to bring to VR or AR. 

This is the wrong question. It assumes a medium can be transcluded in another. It 
assumes that because VR or AR or here XR for short has been named “The Ultimate Display” 
in 1965 Ivan Sutherland, it could somehow capture all past displays, and their formats, 
meaningfully. 

Even though XR eventually could, we are not actually watching movies today that are 
sequentially showing pages of books. Rather we are getting a totally new experience that is 
shaped by the medium. 

So yes, today, we can take a PDF and display it in XR, showing page after page as just 
images at first and try to somehow reproduce the experience of reading in a headset. It could 
open up a lot of new usages because, unlike with a television or screen we can actually 
interact back. We can write back on the content being displayed. Yet, what is the very reason 
for a PDF to exist? A PDF or Portable Document Format exists to be the same on all devices. 
It is a format used not be interacted with but rather be displayed untouched, verbatim. It has 
been somehow modified recently to allow the bare minimum of interaction, i.e signature, 
while remaining integrity for the rest of the document. This has tremendous value but begs 
the question, why would one want this in a spacial world? What is the value of a document 
keeping its shape, namely A4 or Letter pages, while the entire world around it can be freely 
reshaped? What is the value of a static document once interactive notebooks allowing one to 
not just "consume" a document but rather play with it, challenge it, share it back modified? 

PDF does provide value but the value itself comes from a mindset of staticity, of 


permanence, of being closed. 


The reality of most of our daily life, our workflow, is not that static. A document might be 
read printed in A4 or Letter yes but it might just as well be read on a 6.1" portrait display to 
an A4-ish eink device to a 32" 4K landscape monitor. Should the document itself remain the 
same or rather should its content adapt to where and how one wants to consume and 


eventually push back on it? 
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I would argue that any content that is not inviting annotation or even better the actual attempt 
at existing in its target context is stale. Beyond that it is not promoting hermeneutics or our 
own ability to make sense of it. Rather, it presents itself as the "truth" of the matter, and it 
maybe very well be, but unless it can be challenged to be proven as such, it is a very poor 


object of study. 
Consequently a PDF, like a 4.25 x 6.87" paperback is a but a relic of an outdated past. It 


is an outdated symbol of knowledge rather than a current vector of learning. 


The very same content could using HTML provide the very same capabilities and more. 
An HTML page can be read on any device with a browser but also much beyond. An HTML 
page with the right CSS, or cascading stylesheets, can be printed, either actually printed to 
paper or virtually to a document, including a PDF or an ePub, and thus become something 
static again. With the right stylesheets that document could look exactly like the author wants 
on whatever devices they believe it would be best consumed yet without preventing the 


reader from consuming it the way they want, because they have a device nobody else has. 


So even though HTML and PDF can both be brought within XR, one begs for 
skeumorphism. The PDF is again, by what it claims to be its intrinsic value, trapped in a 
frame. Bringing that frame in XR works of course but limits one can interact with it. 
Consequently focusing on bringing PDF to XR means limiting the ability to work with text. 
HTML, especially when written properly, namely with tags that represent semantics rather 
than how to view the content, insure that this is properly delegated to stylesheets is not 
trapped in skeumorphism. The content from an HTML document, in addition to being 
natively parseable by browsers that are already running on XR devices, can then be shapped 
to the usage. It can also be dynamic, from the most basic forms to image maps to 3D models 
that can in turn be manipulated in XR to, last but not least, computational notebooks. While 
PDF are static in both shape and execution model, namely none, an HTML document can 
also embed script tags that can modify its behavior. That behavior allows the intertwining of 
story and interaction. The content then is not just a passive description delegating, poorly as 
argued before due to the minimum ability to modify it while reading it, the interpretation to 
the reader but practically makes the exploration of complex system impossible. An HTML 
document in contrast can present the content so that the system itself being studied can be 
embedded and thus run, not through the mind of the reader, but actually run. The simulation 
become the content letting the reader become an explorer of that content and thus able to try 
to understand much richer and complex systems while confronting their understanding to the 


truth of that system. 


Unfortunately even though there exists today a solution for true responsiveness of 2D 


content, namely stylesheets, this is not true of 3D content, even less spacial content that could 
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be manipulated in VR or AR or both. True responsiveness remains challenging because 
interactions are radically different and the space in which one has such interactions are also 
radically different. A 6.1" portrait display, an A4-ish eink device or a 32" 4K landscape 
monitor are still in the end flat surfaces one can point at, scroll within, etc. Reconsidering this 
and more in both a physical room and a virtual one, eventually with some understanding (e.g 
flat surface detection for floor and walls), leads to a richness of interactions vastly different. 
Consequently one must not just consider how to reflow a 2D document from a rectangle to 
another rectangle but rather to a partly filled volume. Currently there is no automated way to 
day so beside display skeumorphically the document in the volume. This works but is not 
particularly interesting, the same way that one does not watch a movie showing pages of a 
book, even a good book. Instead, being serious about picking a document format, being PDF, 
HTML, ePub or another, means being serious about the interactions with that document and 


the novel interactions truly novel interfaces, like VR and AR, do bring. 


Assuming one still does want to bring 2D documents to a volume, the traditional 
question of provenance remains. As we bring a document in, how does the system know what 
the document is, its format in order to be displayed correctly but also its origin and other 
metadata? The Web did solve most of that problem through URIs and more commonly URLs, 
or DOI being looked up to become URLs pointing to a document, either a live one or the 
archive of one. The Web already provides a solution to how the content itself can move, e.g 
redirection, and browsers are able to follow such redirection to provide a pragmatic approach 


to a digital World that changes over time. 


The question then often becomes, if formats already exist, if provenance can be solved, 
is there not a risk to point only to live documents that can become unaccessible? That is true 
but unfortunately death is a part of life. Archiving content is a perpetual challenge but it 
should not come at the cost of the present. For that still though mechanisms are already in 
place, namely local caching and mirroring. Local caching means that once a document is 
successfully accessed the reading system can fetch a complete or partial copy then rely on it 
in the future if the original document is not available. PWA or Progressive Web Applications 
feature such a mechanism where the browser acts as a reader of documents but also a 
database of visited pages, proxying connections and providing a fallback so that even while 
offline, content that is already on the device remains accessible. Finally mirroring, centralised 
or not, insure that documents do remain accessible if the original source is not available for 
whatever reason. The fact that most websites do not provide either PWA or downloadable 
archives for efficient mirroring is in no way a testimony that the Web does not have the 
capacity for resilience, only that good practices for providing documents over time are not yet 


seen as valuable enough. Luckily efforts like the Internet Archive do mirror content even 
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while the original owner has made no effort to make their content more resilient. Finally 
technical solutions like IPFS, or the InterPlanetary File System, make replication across 
machines more convenient and thus more reliable, again despite more authors not putting the 
necessary care into having their work remaining available beside providing them to a third 
party that will archive without necessarily facilitating access. 

Finally, being PDF, HTML, ePub or another format, the focus hitherto has been on 
bringing text, thus 2D, even arguably 1D if seen as a single string, to a volume, thus a 3D 
space with, i.e AR, or without, i.e VR, context. Even though this provides a powerful way to 
explore a new interface, XR, we must remain aware that this is still a form of transclusion. 
We are trying to force old media in a new one and thus will remain a limited endeavor. Yes it 
would surely be interesting to bring the entirety of Humanity's knowledge to XR but is it 
genuinely a worthwhile pursuit? Past media still exist alongside XR and thus allow use, either 
while using XR (e.g using a phone or desktop screen while using AR or a collaborative 
experience with one person in VR and another video calling from a museum) or before and 
after it (e.g using a desktop to prepare a VR space then share it after) ... or even through our 
memory of it. Consequently even without any effort of bringing the content in XR, it does 
remain accessible somehow. The question rather could become, what native to 3D format 
could better help to create novel usages, based or not on older format. For this there are 
already countless solutions as 3D software long predates XR. That said 2 recent formats did 
emerge, i.e gITF or USD, Graphics Language Transmission Format and Universal Scene 
Description. Both are roughly equivalent but glTF, beside relying on the most popular Web 
format for data, namely JSON, already provides community extensions. This I believe is the 
most interesting aspect. gITF does not try to be encompassing but rather provide the 
minimum feature set then one can build on it for their own usage. That means there is an 
escape valve allowing to be readable by all other software but if one does find it insufficient 
can build on it and adapt it to their needs. This means glTF could become a format not just to 
exchange 3D models to display manipulable objects in XR but finally that such objects could 


address the points touched on before, namely text as a primitive, its provenance explicit. 
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The Case Against Books 


{Analysis: https://fabien.benetou.fr/Analysis/Analysis} 


Books are amazing. Books are compact affordable ways to help Humanity extract itself from 


a naive state of Nature. 


Yet... books are terrible. Books actually were amazing centuries ago. Books are 
symbols of knowledge in the sense that as we look at a book we imagine how it will helps us 
learn. Yet, the truth is far remote from it. Books can be terrible, with poorly written content or 
even arguably worst, beautifully written content is either factually wrong or deceiving. 

Books were once the state of the art of conveing knowledge. That time is long gone, if 
it actually ever existed. Books are terrible because they give the sense of learning. They give 
the impression that because one has read about a topic, they are now knowledgeable about it. 
And yes, imagining that if one knows absolutely nothing about a topic, even the most modest 
book can improve the state of knowledge of that reader. Yet, is it actual knowledge of the 
topic or rather the impression of it? The only way to validate or invalidate that claim is to test 
against reality. The only way to insure that one did learn from a book is to check that newly 
acquired knowledge against the object of the topic itself. That means the reader must not just 
read but rather test. This can be relatively inconvenient, for example of the topic of the book 
is the temperature of the Sun the the reader would need a complex apartus, e.g a spaceship, to 
go and measure. This instead of often delegated to exercises, end of chapters questions with 
answers from the author. The reader instead of reading what the author wrote then have to 
temporarily let go of the book and use their own memory of the content of the book then try 
to see how that knowledge can help solve the challenge. This can be assimilated to a 
simulation, the reader tries to simulate the topic and solve. This already shows a very 
different way to interact with a book then "just" reading. Yet, this leaves much to be desire in 
the sense that the answer provided is often succint. The reader verifies that their answer 
matches the one of the author. If it is correct they assume they know. A great exercise will 
provide ways for the reader to actually verify on their own, like a mathematical proof done 2 
different ways, that the result they find is indeed correct. This though entirely redefine both 
the consumption and creation of a book. At that point a book is not anymore a thing to read 


but rather simultaneously a thing to read and a thing to exercise with. 
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This is a delicate situation for everyone involved. Designing exercise that are genuinely 
bringing the person involved to a better understanding without the ability to correct on the 
way is not the same skill as writing. Also having the confidence in launching oneself in 
exercises is vastly more demanging that reading a sequence of words and assuming they are 
indeed interpreted in a way that the writer would find correct. That means a traditional book 
to read is fundamentally different from what is usualliy refered to as a textbook. Yet, the very 
fact that expensive textbooks are the basis of classes, the one place and moment in time 
dedicated to learning, is not random. Over time the consensus has been that a book itself is 
not sufficient, rather it is a text intertwined with checkpoints that can validate or at least 
invalidate the acquisition of that knowledge that is superior. Most textbooks though are not 
consumed outside of the classroom. This begs the question of why. How come, if a textbook 
is generally regarded as superior, it is limited to a classroom whereas anybody at anytime 


could use it? 


The hypothesis here is that both designing and actually learning from a textbook is 
more demanding than solely reading from a book. Consequently the classroom provide 
support in terms of direct help from the teacher and also motivation from a broader 
curriculum with social markers like a diploma. Yet, textbook in or outside a classroom 
themseves are also relics of the past. For decades now the computer provides a new way to 
both design and consume textbook. Namely that a textbook can now provide not just an 


intellectual environment to run exercises inside of but rather a computational environment. 


A modern text provides the text, the exercises but also the computational environment 
to complete exercises. This sounds like a minor technical improvement but it is a radical 
difference because that environment becomes reality to the reader. The reader now has a 
place, even though an imperfect one in the sense of being simplified, where they can test their 
knowledge. This is a fundamental difference because the reader is not bounded anymore but 
the challenging yet very limited space offered by exercises and their solution. Instead the 
reader can complete carefully crafted exercises but also everything in between. Exercises 
become ways to efficiently navigate through concepts the author believe as essential but 


nothing more. The environment provided is of incredible value to the reader. 


So yes, a book is an amazing device. It has tremendously helped us to progress due to 
compactness and now affordability. Today though a book is not sufficient anymore except for 
the pleasure of reading itself. As a device to improve knowledge the book is outdated. The 
book should instead become computational notebooks providing environments to explore, to 
learn from the reality of the topic. 


Finally, if that is truly the case, how come computational notebooks are not prevalent in 


every field? A simple answer would be that progress takes time and that author of books 
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might not have the skills needed to design computational notebooks. If so, time will 
hopefully solve that issue. A more subtle challenge though might be that the challenge of 
accepting to be challenged through exercises is intelectually and emotionally challenging. It 
requires one to be humble to let reality, even in the form of a simulated one, to push back. It 
always feels easier to assume one know versus discovering that no, truly, one does not. This 
form of interactivity can be seen as a spectrum. From consuming passively a medium, being a 
book to a movie, to consuming it actively while annotating it individually or socially, a form 
of hermeneutics, to finally interacting with the medium itself. That spectrum of interactivity 
might not be solely correlated to the depth of knowledge acquire but also the decision fatigue 
one must go through in order to complete such challenges. 

If computational notebooks should replicate books as the new medium to acquire 
knowledge, we must remain aware of how both designing and consuming them is genuinely 
more demanding to everyone. Hard fun remains hard but the agency it brings to both is a 


truly beautiful prospect for a learning society. 
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Interfaces all the way down 


How prototyping and VR go hand in hand to explore the future of text 


This presentation will explore through one online experience-as-toolkit why interfaces are so 


precious. 


We are navigating our offline and online lives constantly through interfaces. Some are visible 
and explicit like the table of content of books or the API, or Application Programing 
Interfaces, of software libraries while others, like our worldview or virtual reality headsets 


remain implicit and transparent. 


Designing and using interfaces is not trivial and arguably some of the most pressing 
challenge on how to interact with text in all its forms. The experience while showcase its own 
scaffolding in order to invite modifying itself. The objective is, without being fully 
implemented yet, to question if computational notebooks truly are the future of text and if so, 
how if VR is our currently most advanced interface to information can the two become 


coupled to provide the best interface to discovering and sharing knowledge. 
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Stigmergy Across Media 


There is nothing to do to think. One just has to be faced with a random of the countless 
problems we face daily and the brain does its thing, trying to solve it however it can. The 
process seems seemingly transparent, simple even because we just do it, constantly. Yet when 
one has to solve a complex problem, one that arguably does not "fit" in their head, thinking 
takes other forms than an invisible process going through a single head. Thinking extends 
itself through media, being through voices in a heated debate to paper on a poster in an 


academic conference to a research paper or in a computation notebook. 


As we look at the extensions of thoughts, being a printed article, a data visualization, an 
audio recording of a debate, etc we often look at it as a record. That is only partly correct in 
the sense that yes it is a trace of the thought on a medium but it is most than that for the 
author at least. Beyond just a record or a trace, it is a vestige of past live thoughts in the 
making. What it means is that the very action of putting thoughts down on a medium, 
whichever it may be, does help the thinker to think further. 


Feynman reacted with unexpected sharpness: “I actually did the work on the 
paper,” he said. 

“Well,” Weiner said, “the work was done in your head, but the record of it is still 
here.” 

“No, it's not a record, not really. It's working. You have to work on paper and this 
is the paper, Okay?” 

James Gleick 


We must stop limited an artefact to just conveying meaning. We must stop limit the 
perception of an artefact as a way to solely convey meaning but rather always as an 
intellectual stepping stone as it lead to a genuinely new thought that was hitherto impossible 
until then. 

Writing, sketching, programming or waving hands in VR, does not actually matters. It 
is not the preferred medium per se that makes a difference in order to reach furthest thoughts. 
What does matter is actively doing something about the problem on a medium, so stigmergy 


with one self and optionally others. This specific act is extremely powerful creates the 
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potential for us individually and collectively to move forward, wherever we might decide to 
go. 


Author’s original note in email 

I share this because I imagine most people checking the book cover of Drawing Thought 
(Kantrowitz, 2022) would imagine it's about illustration but, just like I was arguing the 
prototype itself doesn't matter, I believe the drawing itself here doesn't matter anymore after, 
only that it lead to a genuinely new thought that was hitherto impossible until then. 

Also I believe drawing, in the case of Kantrowitz, or writing, in the case of Feynman, 
or waving hands in VR for us and others, does not actually matters. What does matter is 
doing something about the problem on a medium, so stigmergy with one self and 
optionally others. This specific act is extremely powerful and as Frode you repeat to us, 
nearly ad nauseam when asking for articles we can then reference, creates the potential for us 


individually and collectively to move forward, wherever we might decide to go. 
Editor’s note 


Also consider Drawing a Hypothesis: Figures of Thought (Gansterer, 2011) and to a degree, 
Lines of thought: Drawing from Michelangelo to now (Riley, Chapman, Seligman, 2016). 
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Journal : Utopiah/visual-meta-append-remote.js 


Not very helpful for publication in a PDF but at least demonstrate a bit how part of the poster 
(or another sliced document) can be manipulated in social VR. Would be better I didn’t let it 
go through the wall or if another avatar was present to better illustrate the social aspect but at 


least it is somehow captured. 


Also here is the code to save back some meta-data, e.g in VR world position, in visual- 
meta in an existing PDF on a remote server https://t.co/y YH9yuSkUs as I noticed the other 
one is in the PDF of the preview of the journal issue. 


It’s challenging to capture it all as its constantly changing but I’m dearly aware of the 
value of it, having traces to discuss on and build back on top thanks to that so precious 


feedback, constructive criticism and suggestion to go beyond. 
~~~~~ code sample ~~~~~ 


const fs = require('fs'); 

const bibtex = require('bibtex-parse'); 

const {PdfData} = require( 'pdfdataextract'); 
const {execSync} = require('child_process'); 
const PDFDocument = require(‘pdfkit'); 
const express = require("express"); 

const cors = require("cors"); 

const PORT = 3000 

const app = express(); 


app.use(cors()); 
app.use('/data', express.static('/')) 


const doc = new PDFDocument(); 


let original ='1.1.pdf 
let newfile ='1.2.pdf 
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let startfile = '/tmp/startfile.pdf 

let lastpage = '/tmp/lastpage.pdf 

let stream = doc.pipe(fs.createWriteStream(lastpage)) 
let dataBuffer = fs.readFileSync(original) 


wee 


var newdata = 


/* client side usage : 

* 

* setup 

* const source = new EventSource(‘https://vmtest.benetou.fr/'+"streaming"); 


source.onmessage = message => console.log(JSON.parse(message.data)); 


* 


* query 
* fetch(‘https://vmtest.benetou.fr/request/test2')then( response => { return 
response.text() } ).then( data => { console.log(data)}) 


*} 


function addDataToPDFWithVM (newdata) { 
PdfData.extract(dataBuffer, { 
get: { // enable or disable data extraction (all are optional and enabled by default) 
pages: true, // get number of pages 
text: true, // get text of each page 
metadata: true, // get metadata 
info: true, // get info 
}, 
}).then((data) => { 
data.pages; // the number of pages 
data.text; // an array of text pages 
data.info; // information of the pdf document, such as Author 
data.metadata; // metadata of the pdf document 
var lastPage = data.text[data.pages-1 ] 
bibRes = bibtex.entries( lastPage.replaceAll("§","")) 
newContent = lastPage.replace("@ {document-headings-end}","@ {fabien- 


220 


test}"+newdata+"@ {fabien-test-end}\n@ {document-headings-end}") 
doc 
//.font('fonts/PalatinoBold.ttf’) 
.fontSize(6) 
.text(newContent, 10, 10) 
.save 
doc.end(); 
execSync(‘pdftk 't+original+' cat 1-r2 output '+startfile) 
stream.on(‘finish', function () { 
execSync(‘pdftk '+startfile+' '+lastpage+' cat output 'tnewfile) 


}) 


sseSend(‘/'+newfile) 


}); 


var connectedClients = [] 
function sseSend(data) { 
connectedClients.map( res => { 


console.log("notifying client") // seems to be call very often (might try to send to 
closed clients?) 


res.write(‘ data: ${JSON.stringify({status: data})}\n\n‘); 
) 


app.get('/streaming', (req, res) => { 


res.setHeader('Cache-Control’, 'no-cache'); 
res.setHeader('Content-Type’, 'text/event-stream'); 
//res.setHeader('Access-Control-Allow-Origin’, '*'); 
// alread handled at the nginx level 
res.setHeader('Connection’, 'keep-alive'); 


res.setHeader('"X-Accel-Buffering’, 'no'); 
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res.flushHeaders(); // flush the headers to establish SSE with client 


res.write(‘data: ${JSON.stringify({event: "userconnect"})}\n\n‘); // res.write() instead 
of res.send() 


connectedClients.push(res) 


// If client closes connection, stop sending events 
res.on('close', () => { 
console.log('client dropped me’); 
res.end(); 
}); 
}); 


app.get("/', (req, res) => { 


res.json('vm test’); 


}); 


app.get('/request/:id', (req, res) => { 
const {id} = req.params; 


console.log(id) 


res.json({"status":"ok"}); 
addDataToPDF WithV M(id) 


) 


app.listen(PORT) 
console.log("listening on port", PORT) 
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Frode Hegland 


The state of my text art + the journey to VR 


At the close of 2022, the year before I expect text in VR (including AR) to take off, I thought 
I should take stock of where my own text systems are and where I plan to go. There are a few 
tweaks I feel are needed in Author, particularly with the Map, some extensions with Visual- 
Meta and minor but useful Reader additions. What has become very apparent over the last 
few months is how hard it has been to envision text in VR. 

Historically the introduction of a new substrate took a while to be taken advantage of. 
This is nothing new. To truly take advantage of a new substrate for, which becomes a new 
textual medium, nothing can replace actual use and experience to inform thinking and 
discussion. We are still struggling to use ‘traditional’ digital media to its full. It is no surprise 
that in the 360, top to bottom, high resolution, powerful computer, high-speed connected 
virtual environment we are still barely scratching the surface. 

For reading, for me, it is about making the experience pleasant. This can be done 
mostly through tradition typography and layout I think. Although text (in the western 
tradition at least) is an operation moving the foveal gaze from left to right, this is not what the 
user has a mental image of, we do not read in the way of a Turing machine. We read with a 
mental impression of the whole document (however weak or strong) and we read with prior 
knowledge. We further read using different points of focus on a page, such as paragraph 
breaks, bolds, and other layouts and so on. 

Basic writing, typing—that is to say text entry—is also good today. I really don’t mind 
what we have today, even the 13” MacBook Pro is pretty great. The way I have polished and 
polished Author for writing, the font styles, the colours and such, have been polished 
primarily for my preference. Others have commented and have their opinions implemented, 
but the software is a testament to what I want for the basics. So yes, this is, to a large extent 
done, in my opinion (for now). 

What I want however, and what I think digital text can afford and XR text can unleash, 
is truly interactive text with flexible views. This is not a new value or vision, it goes all the 
way back to my philosophy of ‘Liquid Information’ and the inspiration of Doug Engelbart’s 
augmentations. Most of what I will describe here can and should be done in traditional 
digital environments, which is what I have been working on doing with Author and Reader. 
Hopefully XR will provide enough curiosity to make it happen and enough interest from 
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then public to make it viable. 


State of the my art 


A few specific interactions in my software Author and Reader I'd like to highlight of the 
way it is at the end of 2022 include: 


Much word processing and reading is quite stilted in my opinion and this is something I try to 

address with my software, to make the process flow better, to make it more liquid. I therefore 

outline some of the interactions currently possible in Author and Reader: 

¢ See only headings by folding the document {cmd-+/-} 

¢ See all the occurrences of Selected Text plus headings (to see where the occurrences are) 
{cmd-F} 

¢ See Summaries of sections plus heading (not implemented yet) 

¢ See all the occurrences of Bold text plus headings (not implemented yet) 


¢ See all the occurrences of author Highlighted Text (currently implemented in Author 
and to be exported via Visual-Meta so that a reader can choose to see author highlights) 
{cmd-+/- in Author} 


When using my software, simply called Liquid, a macOS wide text utility, the following is 
also available easily with point and click and instantly when the basic keyboard shortcuts 


have been learnt: 
¢ Select any text and instantly look it up in any number of resources 


* Select a section of heavy text and have it be opened up with double breaks on period and 


singles breaks on command etc.t (as can also be done with Liquid) 
¢ Select any text and translate 


¢ Select any number and convert 


The innovation needed, in my mind, is primarily with Editing & Research: 
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Editing 


Once I have my basic text down, it becomes a task to extend some sections, shortening others 
and-this is the difficult bit for me—making sure that the flow of the text makes —_ sense and 
different sections relate and that there is a coherent way to read as an overview. 

I want to be able to write an executive summary ‘outline’ of sorts and make the 
document flow from that. A traditional outline is not what I mean however. A Table of 
Contents can be an overview but in a normal long academic paper there is so much in each 
headings section that disappears below the surface. This is a question I continuously grapple 
with. 

To write the kinds of documents I want to write, where the executive summary really 
serves as a Starting point to the whole document and should be self contained as a useful unit 
to read, with supplemental text should be written as ‘units’ of knowledge rather than 


laboriously written long-form text written afresh every time I write something. 
A workflow for this needs to be able to involve both the authoring and the reading. 


This is what I am doing with the ‘Defined Concepts’ in Author, which become exported 


as a Glossary to Reader (in PDF). There is so much more that should be possible though. 


An important side note: 1 am not wedded to PDF but J find its frozen aspect reassuring for 
the long term and with Visual-Meta the metadata is not hidden which should make it more 


useable. 


Defining your concepts as you write & access as you read (what I have started on) 
Imagine continuously and easily defining as you are writing, including the word ‘I’ with 
information about yourself at the time of writing and having this available automatically in 
the future. When someone reads your work they should be able to stay on the surface layer if 
they know enough about you and your work but if they need further information then can 
make use of the definitions you have written, which is safely stored in the document as a 
Glossary. Defining concepts for re-use is the key to my approach to what I see as the 
future of text. 


The reader should be able to choose what to see when reading a document, including access 


to the Glossary in the appendix: 


¢ Select text and cmd-F to see all the occurrences of the selected text and if the text is a 


Defined Term, and also show the Glossary definition on top of the screen, with any other 
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terms in bold so that they can be clicked on to load. (This is possible now in my ‘Reader’ 
application for macOS, hopefully for 10S in 2023) 

¢ See all the occurrences of Defined Concepts in the document plus headings {cmd-shift- 
D} 

¢ See a Map of Defined Concepts to see how they relate, in a visually clutter free format 
{cmd-M in Author} 

¢ See all the occurrences of Names plus headings {cmd-shift-N} 

¢ Glossary definitions after each term in the document, as a hypertext stretchtext 


(currently only a concept since re-flowing PDF documents is very hard) 


A key is this: Less text is better. In order to be able to write less text per ‘document’ we need 
some mechanism to write in a more modular—and well connected—hypertext fashion, and not 


just connected to external sources, but within, hence the attempt to re-invent glossaries and 


endnotes”. It is clear from my experiments in XR that simply having a massive display is not 
the answer, or many large displays, it is still an effort on the side of the software developer 


and the user to decide what goes where and how this changes. 
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Research 


Reading for research is partly about navigating a document for relevance, close reading for 


critical comprehension and to see connections. 


Navigation 


Reading for research needs better ways to navigate the document. We have experimented 
with many ways of doing this where the issue is how much of each section needs to be shown 
for it to be useful and not overwhelming. I have found that simply having an arrow key right 
and left for next and previous —as is normal now-—can be augmented with an arrow key down 
and up, which will take the reader to the next or previous page with a heading, is a good 
solution. The user does not need to spend time analysing every page when scrolling through 
the document and does not need to guess based on a plain table of context, the pages speak 
for themselves. 


Close reading 


Close reading is aided by good typography and layout for basic readability and the basic 
interactions outlined in the section State of the my art above. Further work can of course be 
done here to really elevate the reading experience through giving the user complete and near- 


transparent control over the appearance and interactions of what they read. 


Connections 


It is also important for me to be able to cite easily and that means that within a community, 
such as academic community or our Future of Text community, to be able to cite a document 
and have it open on click if I have on my hard drive/cloud/system), not just a link to a 
website for download or to open a Reference Manager. This is part of the future of Visual- 
Meta. Already Visual-Meta allows for copy and paste to cite, but the reading and following of 
citations needs to be improved. 


Other Perspectives 


Research with highlights by authorities in the field, such as the highlight above. This is social 
annotation but it matters who did the annotating, it can be a DJ and you can choose to cite a 


document with a specific person’s annotations. 
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Xtended Reality for text 


I can so easily imagine a laptop display extending into different displays to let me have at-a- 
glance access to at least the following elements one on each display, as discussed in my 
article on Displays: 

* Table of Contents 

* Map of Defined Concepts 


¢ References 


VR gives us a much wider workspace, which can truly help some with editing and seeing 
connections, both in our own work and in what we reading for research. I think we need to 
start with the basics, allowing for traditional digital documents to be accessible in VR 
environments, with as much metadata robustly attached (of course I suggest Visual-Meta as 
part of the solution to this) and then have the interactions magically grow out of this 
document as our experience and imagination grows. Similarly, those who can imagine 
completely new textual worlds should do so, and in dialog we can realise the actual Future of 
Text. 


Making it happen 


Much of what I plan to do can be done and should be done in 2D but although I have built 
some of it, it’s hard to finance more, partly since there is only a limited curiosity among users 
for different ways to read and write outside the Microsoft Word and Apple Pages paradigm 
and the Google Docs online method. Of course there are brilliant software out there such as 
Literature & Latte Scrivener, iA Writer and The Soulmen’s Mac Ulysses. In my experience as 
a small, independent developer however, it is very hard to break through to actually show 
people another way, which may or may not be to their taste and style. As I highlight in 
several places since I feel it is so crucial, VR gives us an opportunity for renewed 
curiosity. I hope I can make use of this for my own perspective, my own software, and for 


the whole community to get to the next level of text augmentation. 
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Frode Hegland 


The case for books 


Fabien wrote a piece on the case against books and here is my small piece on the case for 
books. 

Books, in my view, are intentionally bound collections of pages which are explicitly 
‘published’ though not necessarily shared with a wider audience, at a specific time. Books are 
also self-contained though they rely on explicit and implicit connections to convey meaning. 

Explicitly published is important since they are not ‘forever documents’ like a Google 
Doc or that Word document manuscript you have languishing in your word processor. They 
are defined as being done, at least for the current version. 

The fact that they are published at a specific time marks them in the history of the 


evolution of ideas and assertions and allow them to be cited and for flexible views to be built. 
Robustness 


Of course books should be able to come in many formats but a basic format of the book is 
that it can be self-contained and therefore, with metadata solutions such as Visual-Meta, can 


contain rich information about the book even it is printed on paper. 
Book Bindings 


The fact that a books are bound is of significance. When books were only physical, the 
physical bounding was not something which could be changed unless the spine was cracked 
or pages photocopied or hand copied. 


Digital Bindings 


Digital bindings should allow the author/publisher to produce an initial binding but the reader 
should also quite easily be able to break the book up and further share, or publish, their 


section of the book (rights pending of course). Their edit of the book into a new binding 
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could be just a single article, a single page or a collection of articles. 


If the book is in a series, such as The Future of Text is, then the user should be able to 


bind it all into one binding, should they wish. 
Or combine different sources into a binding, as a teacher might do with photocopies. 


Further, the user should be able to annotate the bound book as a book ‘DJ’ of sorts, 


where people might even subscribe to get that persons’ views of books. 


And there you have it. We should not only share information as books or even journals or 
magazines, but books do have their place and I suspect always will, but their utility will 


change with what the technologies make possible. 


Future Books 


There is no reason books need to stay rooted in the past, they can be set free with increasing 
technological opportunities. We are only just beginning to imagine books which have special 
characteristics in VR, without being locked into only being readable in VR. 
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Frode Hegland 


‘Just’ more displays? 


At the close of 2022 when the Quest 2 has become quite popular, the Quest Pro has just been 
released (I’ve used mine for one day so far) and we are all expecting the Apple HMD early 
next year, a comment is see every once in a while is that XR should be’ more than just more 
displays’. This is because it is relatively easy it seems to use a HMD as a receiver of a 
computer’s display information taking over the main display and adding more ‘virtual’ 
displays when needed. The implication is that this is simply too easy and does not take good 
advantage of what VR has to offer. As a huge fan of the potential of VR, I disagree. Yes, it 
might very well be technically easy and yes, the future will bring truly new dimensions to 
VR, there is no question in my mind. However, let’s not bury what it useful just because it 


is easy to build—not everything has to be a demonstration of technical prowess. 


A key issue is that text is hard to read when it does not have a clear and plain 
background. This is why text floating as a hologram in sci-fi looks cool but is not practical 
to work with. When you have a background you in effect have a screen. And that’s ok. It does 
not have to be a regular sized screen, it could be a magically resizable screen which can go 
anywhere and be moved anywhere without physical effort. Perhaps most importantly, eye 
tracking can allow screens to fade away when not needed. This can mean that the user can 
have the best of a focused writing experience-ore reading experience—but the user can look to 


the sides and supplemental information appears—without being intrusive. 


Displays/floating windows of any size which can be accessed and removed at a glance, is 
huge. 


The thing is, the way screens currently works is that it is the computer which generate extra 
screens for the HMD to access and display, not the applications. To have instant integration 
with VR/AR, the windows should be on an application basis or created through Web VR 
for extra screens on demand. These screens should also be addressable by the host software 


for display sizing and show/hide (based on eye tracking, gesture or other). 
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This would allow me as a developer to have my software almost instantly available in 

VR and AR in a more useful form. Both my Author word processor and my Reader PDF 

viewer. I would simply add a function to the software to allow for the creation of such extra 

displays and then voila, the user will have a much more useful workspace in VR. 

* On the left, for example, a table of contents could appear when glancing left. 

* On the right, for example, could be all my available citations. 

¢ Aconcept map could appear on the wall opposite the user when the user wishes to view it 
(which might be all the time of course). Flexible displays, both small and large, which are 
aware of each other (same software running them) can help developers quickly port to 
HMD’s. 


Hardware developers, build this as an easy to access API and us developers will come. 
This can be much more powerful than what we saw 20 years ago in Minority Report. More 


human scale, more useful and almost instantly available to developers to use. 


Minority Report. Anon, 2002. 
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Stepping out 

Further interactions can be extended to have objects from within the displays be pulled out, 
where they exist in the AR space as a flexible, 3D shaped display for their contents, such as 
photographs etc. Dragging text out of a display could make it float as a clipping, with a 
memory of where it comes from. 

If this can work, then it would be great to allow for gestures to work to modify the 
contents of the displays—maybe-since the user will already be on keyboard and mouse/ 
trackpad. What will definitely be useful will be to allow the user to effortlessly modify where 
the displays are and their sizes. For example grab the display by one (or two) vertical sides to 


move it. Grab by top or bottom to rotate on the x-axis and grab by corners to re-size. Simple. 


Size matters 

What testing showed however, is that while multiple and large screens add a powerfully 
useful dimension, interactivity will still need to be designed to make it useful. For example, 
in the screenshot on the next page is the Table of Contents of my thesis, you can see it is 


much too tall to be readable from a single head position. 


* On the left is only the level one headings and a few highlighted pieces of text. 


* On the right is the full table of contents with the level one shown in the same size, to show 


the massive scale. 


This indicates that it’s great to have mod displays but with ‘infinite’ scale we can easily 
surpass human scale and therefore we will need interactions to help us define the view 
flexibly. 


233 


ABSTRACT 

Declaration of Authorship 
Acknowledgements 
Introduction 


Background 
Foc ny own software development can we e-ink seem text on connected computer seems? 


The Design of Visual-Meta 

Visual-Meta Specification 

Implementation of Visual-Meta in Augmented Text Tools 

validation Survey 

Expert Interviews 

HIGHLIGHT MORE ACADEMIC - as XXX - because XXX there? - Future: Doug was in world of nothing 96 
‘exciting. Paradigms entresched! Now we have VR! - This can simply bea shenmary, 

Impact 


Conclusion 


Appendix A : Full Scoping Survey 
Appendix B : Full Visual-Meta Structure 
Appendix C : Full Evaluation Survey 
Appendix D : Expert Interviews 
Appendix E : App Store Reviews 
Appendix F ; BBC Interview Transcript 
Appendix G : Future of Text Testimonials 
Appendix H : ‘The Future of Text Redux’ 
A question of Governance 


To The Future 


234 


Fabien Benetou responds 
On the notion of windows by the application: That exists. This is not "just" a potentially good 
idea anymore : I tried one 3 years ago https://twitter.com/utopiah/status/ 
1164059349490249728 and a bit later again with much more demanding content https:// 
twitter.com/utopiah/status/1261753 166321909760 

It has been funded by Valve and is open source https://gitlab.freedesktop.org/xrdesktop/ 
xrdesktop 

What's interesting also is to put this back in perspective. This was already implemented 
in 2014 https://twitter.com/utopiah/status/1560500042963771392 as Motorcar that I 
discovered.... while trying another open source VR window manager https://twitter.com/ 
utopiah/status/15606072023 14174465 , namely SimulaVR https://github.com/SimulaVR/ 
Simula/ 

My point here is obviously not to criticize the idea but rather to focus on the gaps of 
these existing solutions. 

These are desktop windows managers for desktop VR. They take existing windows, e.g 


text editor or video player, and let you organize them in space. 


For you to try them you'd need a desktop computer with a relatively powerful GPU running 


Linux then connect your headset, Quest 2 or Quest Pro, to it. 


Frode Hegland responds 


Thank you Fabien, this is great to see. If it could be transparently available to desktop 
software developers for use in VR that would be a huge step. I am happy that it technically 
works though. 
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Frode Hegland 


Page to Page Navigation 


Originally email to group: 

There are different issues when reading a document for navigation. One issue is that you 
simply want to skip to the next heading since you are done with where you are and there are 
many pages of text before the next heading for you to skip through—judging all of them on the 


way to see when the next heading appears-to find if the text section is worth reading. 


I have made three brief tests using our book as example. 


The issue is how to let a user jump around our book in a convenient way. 


¢ A table of contents is useful but only if you know the author of the section (as in the case of 
our book) or if the title is very clear, which is rare. 
¢ A table of contents with tags/further metadata is hard to establish and can get messy, though 


this is definitely worth further investigation, particularly in VR/AR environments. 


* It therefore seems to me that the ability to jump to the next or previous heading, not just 
page, is of use. Several of our articles are very long, so if you are not interested in one you 
should not have to click or gesture multiple times. The short video below shows a test on this 
basis. In that video’s description there is a link to a version of only one page, rather than two 


page spread, and also a continuous scroll test. 
https://youtu.be/6hnr0jwT4kM 


Anyway, thanks for looking at this. The point of doing these is simply that in VR/AR we are 
not free of all constraints, we have different constraints, which is taking us time to learn as 
we explore the environment. The potential is vast, and we are just calling our way to greater 
understanding. We should do fully interactive tests of course, but we should also do tests like 
this which is simply a mock-up of our book where I deleted all the pages which didn’t have 
headings on them. I think that this will be useful for Reader in 2D and might work in VR, but 


I don’t know. I hope there will be tests or mockups or presentations or pencil sketches from 
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whomever is interested, because otherwise we stay discussing abstract generalities and that is 
worthwhile, but we should also try to be grounded in this new reality, and on the way 


discover what ‘grounded’ actually means. 


Adam Wern responds: 

Heading-by-Heading (and sub-heading) navigation is standard in many PDF readers (like 
Preview on Mac). Works with any PDF that has a proper ToC. That's how I usually read 
longer non-fiction PDF books. 


And you can still scroll or navigate page-by-page with [Space] 


Frode Hegland responds: 


Yes and thanks for showing me this, it is indeed the same principle. However you need to 
have the focus in the table of contents, which is different since the table of contents 
needs to be visible. What I propose is that down arrow always goes to next heading, no 
matter what view and right arrow next page (and in reverse of course the opposite). Either 
way, the metadata for headings needs to be present, which it rarely is, but great when it is. 
This is of course a Visual-Meta issue for me and Reader should also support ‘native’ PDF 
headings. 


Adam Wern responds: 

Yeah, a digital ToC without interactions is sad. Another takeaway is that we should always 
include regular PDF ToC:s for books as can help millions of readers directly without any 
special software (and also screen readers). I've noticed that more recent academic texts 


include a ToC as well, which is excellent 
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Journal : Academic & Scientific Documents in the Metaverse 


Recall the world before it all became digital. You are in a meeting where you have a 
printout of a relevant document and a notepad. You underline relevant parts of the document, 
you write notes and draw diagrams in your notepad. You are also given a stack of index cards 
so that you can all do some brain-storming and those cards are pinned to a wall and moved 
around as you discuss them as a group. The facilitator even pins a few lines of string between 
related cards. You take a picture of this and since you don’t need the document you printed 
out—since the meeting went so well—you fold it into a paper airplane and fly it into the bin. 
Now picture yourself in a fully digital environment where you have the same document 
and notepad and you use systems like Google Docs to collaborate and even a projector or a 
big screen for the cards to be put up and moved around by the facilitator. This is pretty much 
the office life many of us live in today. You can’t exactly fly the airplane to the bin, you have 
given up arbitrary interactions for those which are more useful in a work environment, such 
as the ability to instantly edit and share your information. Every environment you work in 
will of course have tradeoffs as to what you can do there. So let’s go to the near-future and 
don our AR/VR headgear and enter a meeting in the Metaverse with the same document 
and a notepad, in richly interactive knowledge room. You will now be able to do magical 


things, as we can dream about today, and even build demos of: 
¢ You can spread the document out in and have it float in the air where you want it to. 


¢ Any included diagrams can be pulled out and enlarged to fill a wall, where you can 


discuss it and annotate it. 


¢ Any references from that document can be visualised as lines going into the distance and a 


tug on any line will bring the source into view. 


¢ You can throw your virtual index cards straight to a huge wall and you and the facilitator 


can both move the cards around, as well as save their positions and build sets of layouts. 
¢ Lines showing different kinds of connections can be made to appear between the cards. 


¢ Ifthe cards have time information they can also be put on a timeline, if they have 


geographic information they can be put on a map, even a globe. 


¢ If there is related information in the document you brought, or in any relevant documents, 


they can be connected to this constellation of knowledge. 


What you can do is only limited by our imagination and the tools provided. And it is 
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also limited by the enabling infrastructures. What you cannot do is leave the room with 
this knowledge space intact. The actions you can perform on the knowledge elements in the 
room is entirely predicated by the ‘affordances’ the room gives you, to use a term from 
psychology which is also used for human-computer-interaction. It is akin to taking a picture 
from one picture editing program to another program—even though it’s the same picture, you 
cannot expect to be able to perform the exact same functions—such as special photographic 
filters. The difference in the metaverse will be that the entire environment will be software, 
both the visual aspects of the environment and the interactions you will have, and that means 
it will be owned by someone. Meta owns everything you do in their Quest headsets when in 
their environments, such as Horizon Workrooms, you cannot perform operations which they 
have not made possible through programming the space they own. Apple and Google will try 
to own the knowledge spaces they provide as well. Consider just a few documents: 
Currently you cannot fully open a document into a VR space, you can either view your Mac 
or Windows computer screen or you can have the document as sheets, but let’s skip ahead to 
when you can indeed open the document and its metadata is available to you. You open a 
document in the knowledge space and you: 
¢ Pull the table of contents to one side for easy overview. 
¢ Throw the glossary into another part of the room. 
¢ Throw all the sources of the document against a wall. 
¢ You manipulate the document with interactions even Tom Cruise would have been jealous 
of in Minority Report‘. 
¢ You read this new document with the same interactions and decide to see the two 
documents side by side with similarities highlighted with translucent bands, Ted Nelson 
style. 
Then you have a meeting and you have to leave this knowledge room. Your next meeting is in 
a different type of room developed by a different company but the work you have just done is 
so relevant to your next meeting so you wish you could take across the work you have done 
but you cannot. The data for how the information is displayed and what interactions you 
can do are determined by the room you are in, since that is the software which makes 
the interactions possible. What we need is to develop open standards for how data, in the 
form of documents but also all other forms of data, can be taken into these environments and 
for how the resulting views, which is to say arrangements, of this information is stored and 
handled. How will the stored, how will it be accessible and who will own it? This will be 


for us to decide, together. Or we can let commerce fence us in. 
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Jack Kausch 


Why We Need a Semantic Writing System 


Can there be non-sequential text? 
The Greeks thought Egyptian hieroglyphs were allegorical icons which conveyed pure ideas. 
This interpretation was passed down to the Renaissance, and combined with misconceptions 
about Chinese language. In the early modern period, Europeans dreamed of creating a 
universal pictographic language which, combined with an encyclopedia, would translate all 
knowledge into every language in the world. 

We now know that Egyptian hieroglyphs are not just pictures. They also convey sound. 
The boundary between pictographic proto-writing and what we consider writing with a 
grammar is the Rebus Principle, where a picture begins to stand for a sound by a process of 
visual punning. This was practiced in an extreme form in early Egyptian history, and gave 


rise to the multi-layered nature of the writing system. The best term to describe writing 


Ze ce. 


systems like this is not “logographic” or “ideographic” but the Mandarin J% a “xing shéng”, 
which roughly translates to “phonosemantic.” 

Both Cuneiform and Egyptian have the quality of conveying spoken speech alongside 
semantic classifier symbols, which disambiguate transcriptions. The convention for how to 
read Hieroglyphs is not justified against any one direction on the scroll or stela, but follows 
the rule to read “into the faces of animals” or in the opposite direction that all the characters 
are looking. Thus hieroglyphs can be read from right to left, left to right, top to bottom, and 
vice versa, depending on how they are written. 

However: every inscription is still sequential. Even boustrophedon texts from the early 
Greek period, which reverse direction every line, continue to convey language linearly. The 
reason for this is that speech, while continuous, is sequential, and text encodes speech. Text 
takes continuous phonological features and represents them as discrete symbols, yet the 
content of the representation remains sound-based. There is not, and has never been, a “non- 


discursive” writing system, like the Greeks once thought about Egyptian hieroglyphs. 
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This is not to say that there is not great value in pictographic systems of representation which 
have no relation to language, such as Emoji. It is just that they are not considered writing 
because they have no phonological content, and as such they do not represent the grammar of 
natural language. Birchbark scrolls such as the Ojibwe wiigwasabak or the Mi'kmaq 
hieroglyphs can convey complex layers of narrative meaning, but their interpretation is 
limited to those already initiated into an oral tradition. What we consider text remains a 


function of what is speakable. 


We are entering an era that wishes to challenge the linearity of text. The distributed 
nature of the Web, and the “horizontal” potential of hypertext to link documents together, 
seems to invite a world in which the sequential nature of the printed book is altered. What 
this change amounts to is another transformation in documentation. The codex made very 
different social modes of organization possible from the scroll (indeed, it may have been 
partly responsible for the rise of Christianity) and printing transformed the relations between 
individuals and the book. The nature of documents, including how they are stored and 
disseminated, will now inevitably change. 

There is a limit, however, to how non-sequential we can make text in its own right, for 
the reasons discussed above. Emoji appear to offer an interesting alternative, yet for all their 
expressive power, like most pictographic symbol sets, they remain ambiguous. Icons provide 
an ability to convey certain kinds of information, and even establish natural classes. We 
encode them with the same standards as text, and they are treated as text-like entities. Yet 
metaphorical combinations of icons can have many interpretations, and there are too many 
things in the world to create an icon for every one. There is thus no small inventory of icons 


which will satisfy the constraint of being able to combine them into every possible concept. 


Our new tools have nearly endless potential for the representation of mathematical, 
particularly geometric, entities. Text on the other hand is dependent on standards which 
encode individual characters, and in turn influence how the text is formatted, and what 
interfaces can be made for users to work with it, i.e., to read and write. This is foreign to our 
visual interfaces, whether phones or monitors, which, composed of pixels, are ideally suited 
to displaying graphics and shapes. 

To return now to the European dream of a universal character language from the 
Enlightenment: where such a writing system is similar to emojis and geometry, it loses many 
of the characteristics we ascribe to text, because it transcends the limits of language. It is non- 
sequential, but it is too vague to consistently convey the writer’s intent. Where such a writing 
system conveys linguistic and grammatical information, it is constrained by the phonological 
traits of each language, and cannot be said to be “universal.” This is the conventional text we 


already have. 
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The answer is probably somewhere in between, similar to what the Egyptians 
discovered all those years back during the period between the reign of Mena-Narmer and 
Djoser. Some combination of sounds and meanings could serve as a mnemonic device to 
clarify both categories, and potentially integrate well into current speech synthesis 
technology. If there can be non-sequential text it will be found at the intersection of the visual 


image, geometry, well-formed semantic logic, and phonological natural language. 
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Jad Esber 


Monthly Guest Presentation : 21 February 2022 
Video: https://youtu.be/i_dZmp59wGk?t=513 


Jad Esber: Today I'll be talking a little bit about both, sort of, algorithmic, and human 
curation. Ill be using a lot of metaphors, as a poet that’s how I tend to explain things. The 


presentation won’t take very long, and I hope to have a longer discussion. 


On today’s internet, algorithms have taken on the role of taste-making, but also the 
authoritative role of gatekeeping through the anonymous spotlighting of specific content. If 
you take the example of music, streaming services have given us access to infinite amounts 
of music. There are around 40,000 songs uploaded on Spotify every single day. And given the 
amount of music circulating on the internet, and how it’s increasing all the time, the need for 
compression of cultural data and the ability to find the essence of things becomes more focal 
than ever. And because automated systems have taken on that role of taste-making, they have 
a profound effect on the social and cultural value of music, if we take the example of music. 
And so, it ends up influencing people’s impressions and opinions towards what kind of music 
is considered valuable or desirable or not. 

If you think of it from an artist’s perspective, despite platforms subverting the 
power of labels, who are our previous gatekeepers and taste-makers, and claiming to 
level the playing field, they’re creating new power structures. With algorithms and 
editorial teams controlling what playlists we listen to, to the point where artists are so 
obsessed with playlist placement, that it’s dictating what music they create. So if you listen to 
the next few new songs that you hear on a streaming service, you might observe that they’ Il 
start with a chorus, they’ll be really loud, theyll be dynamic, and that’s because they’re 
optimising for the input signals of algorithms and for playlist placement. And this is even 
more pronounced on platforms like TikTok, which essentially strip away all forms of human 
curation. And I would hypothesise that, if Amy Winehouse released Back in Black today, it 
wouldn’t perform very well because of its pacing, the undynamic melody. It wouldn’t have 
pleased the algorithms. It wouldn’t have sold the over 40 million copies that it did. 

And another issue with algorithms is churning standardised recommendations that are 
flattening individual tastes, they’re encouraging conformity and stripping listeners of social 
interaction. We’re all essentially listening to the same songs. 
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There are actually millions of songs, on ‘Spotify’, that have been played only partially, 
or never at all. And there’s a service, which is kind of tongue-in-cheek, but it’s called 
‘Forgotify’, that exists to give the neglected songs another way to reach you. So if you know 
are looking for a song that’s never been played, or hardly been played, you can go to 
‘Forgotify’ to listen to it. So, the answer isn’t that we should eliminate algorithms or machine 
curation. We actually really need machine and programmatic algorithms to scale, but we also 
need humans to make it real. So, it’s not one or the other. If we solely rely on algorithms to 
understand the contextual knowledge around, let’s say, music, that’1l be impossible. Because, 
at present, human effort, popularity bias, which means only recommending popular stuff, and 
the cold start problem is unavoidable with music recommendation, even with very advanced 
hybrid collaborative filtering models that Spotify implies. So pairing algorithmic discovery 
with human curation will remain the only option. And with human curation allowing for the 
recalibration of recommendation through contextual reasoning and sensitivity, qualities that 
only humans really can do. Today this has caused the formation of new power structures that 
place the careers of merging artists, let’s say on Spotify, in the hands of a very small set of 


curators that live at the major streaming platform. 


Spotify actually has an editorial team of humans that adds context around 
algorithms and curates playlists. So they’re very powerful. But as a society, you 
continuously look to others, to both validate specific tastes, and to inspire us with new tastes. 
If I were to ask you how you came up discovered a new article or a new song, it’s likely that 
you have heard of it from someone you trust. 

People have looked to tastemakers to provide recommendations continuously. But part 
of the problem is that curation still remains an invisible labour. There aren’t really 
incentive structures that allow curators to truly thrive. And it’s something that a lot of 
blockchain advocates, people who believe in Web3, think that there is an opportunity for that 
to change with this new tech. But beyond this, there is also a really big need for a design 
system that allows for human-centred discovery. A lot of people have tried, but nothing has 
really emerged. 

I wanted to use a metaphor and sort of explore what bookshelves represent as a 
potential example of an alternative design system for discovery, human-curated discovery. 
So, let’s imagine the last time you visited the bookstore. The last time I visited the bookstore, 
I might have gone in to search for a specific book. Perhaps it was to seek inspiration for 
another read. I didn’t know what book I wanted to buy. Or maybe, like me, you went into the 
bookstore for the vibes, because the aesthetic is really cool, and being in that space signals 
something to people. This book store over here is one I used to frequent in London. I loved 


just going to hang out there because it was awesome, and I wanted to be seen there. But 


246 


similarly, when I go and visit someone’s house, I’m always on the lookout for what’s on their 
bookshelf, to see what they’re reading. That’s especially the case for someone I really admire 
or want to get to know better. And by looking at their bookshelf, I get a sense of what they’ re 
interested in, who they are. But it also allows for a certain level of connection with the 
individual that’s curating the books. They provide a level of context and trust that the things 
on their bookshelves are things that I might be interested in. And I’d love to, for example, 
know what’s on Frode’s bookshelf right now. But there’s also something really intimate about 
browsing someone’s bookshelf, which is essentially a public display of what they’re 
consuming or looking to consume. So, if there’s a book you’ve read, or want to read, it 
immediately triggers common ground. It triggers a sense of connection with that individual. 
Perhaps it’s a conversation. I was browsing Frode’s bookshelf and I came across a book that I 
was interested in, perhaps, I start a conversation around it. So, along with discovery, the act 
of going through someone’s bookshelf, allows for that context, for connection, and then, the 
borrowing of the book creates a new level of context. I might borrow the book and kind of 
have the opportunity to read through it, live through it, and then go back and have another 
conversation with the person that I borrowed it from. And so recommending a book to a 
friend is one thing, but sharing a copy of that book, in which maybe you’ve annotated the text 
that stands out to you, or highlighted key parts of paragraphs, that’s an entirely new 
dimension of connection. What stood out to you versus what stood out to them. And it’s 
really important to remember that people connect with people at the end of the day and not 
just with content. Beyond the books on display, the range of authors matters. And even the 
effort to source the books matters. Perhaps it’s an early edition of a book. Or you had to wait 


in line for hours to get an autographed copy from that author. 


That level of effort, or the proof of work to kind of source that book, also signals how 
intense my fanship is, or how important this book is to me. 

And all that context is really important. And what’s really interesting is also that the 
bookshelf is a record of who I was, and also who I want to be. And I really love this quote 
from Inga Chen, she says, “What books people buy are stronger signals of what topics are 
important to people, or perhaps what topics are aspirationally important, important enough to 
buy a book that will take hours to read or that will sit on their shelf and signal something 
about them.” If we compare that to some platforms, like Pinterest for example. Pinterest 
exists to not just curate what you’re interested in right now, but what’s aspirationally 
interesting to you. It’s the wedding dresses that you want to buy or the furniture that you want 
to purchase. So there’s this level of, who you want to become, as well, that’s spoken to 


through that curation of books, that lives on your bookshelf. 


I wanted to come back and connect this with where we’re at with the internet today and 
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this new realm of ownership and people are calling social objects. And so, if we take this 
metaphor of a bookshelf and apply it to any other space that houses cultural artefacts, the 
term people have been using for these cultural artefacts is social objects. We can think of, 
beyond books, the shirts we wear, the posters we put on our walls, the souvenirs we pick up, 
they’re all, essentially, social objects. And they showcase what we care about and the 
communities that we belong to. And, at their core, these social objects act as a shorthand to 
tell people about who we are. They are like beacons that send out the signal for like-minded 
people to find us. If I’m wearing a band shirt, then other fans of that artist, that band will, 
perhaps, want to connect with me. On the internet, these social objects take the form of 
URLs, of JPEGs, articles, songs, videos, and there are platforms like Pinterest, or Goodreads, 
or Spotify, and countless others that centre around some level of human-curated discovery, 
and community around these social objects. But what’s really missing from our digital 
experience today is this aspect of ownership that’s rooted in the physicality of the books on 
your bookshelves. We might turn to digital platforms as sources of discovery and inspiration, 
but until now we haven’t really been able to attach our identities to the content we consume, 
in a similar way that we do to physical owned goods. And part of that is the public histories 
that exist around the owned objects that we have, in the context that isn’t really provided in 
the limited UIs that a lot of our devices allow us to convey. So, a lot of what’s happening 
today around blockchains is focused on how can we track provenance or try to verify that 
someone was the first to something, and how do we, in a way, track a meme through its 
evolution. And there are elements of context that are provided through that sort of tech, 


although limited. 


There is discussion around ownership as well. Like, who owns what, but also 
portability. The fact that I am able to take the things that I own with me from one space to 
another, which means that I’m no longer leaving fragments of my identity siloed in these 
different spaces, but there’s a sense of personhood. And so these questions of physical 
ownership are starting to enter the digital realm. And we’re at an interesting time right now, 
where a lot of, I think, design systems will start to pop up, that emulate a lot of what it feels 
like to work, to walk into a bookstore, or to browse someone’s bookshelf. And so, I wanted to 
leave us with that open question, and that provocation, and transition to more of a discussion. 


That was everything that I had to present. 


So, I will pause there and pass it back to Frode, and perhaps we can just have a 


discussion from now on. Thank you for listening. 
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Dialogue 


https://youtu.be/i_dZmp59wGk?t=1329 


Frode Hegland: Thank you very much. That was interesting and provocative. Very good for 
this group. I can see lots of heads are wobbling, and it means there’s a lot of thinking. But 
since I have the mic I will do the first question, and that is: 

Coming from academia, one thing that I’m wondering what you think and I’m also 
wondering what the academics in the room might think. References, as bookshelf, or 
references as showing who you are, basically trying to cram things in there to show, not 
necessarily support your argument, but support your identity, do you have any comments on 
that? 

Jad Esber: So, I think that’s a really interesting thought. When I was thinking of 
bookshelves, they do serve almost like references, because of the thoughts and the insights 
that you share. If you’re sitting in the bedroom, in the living room, and you’re sharing some 
thoughts, perhaps you’re having a political conversation, and you point at the book on your 
shelf that perhaps you read, that’s like, “Hey, this thought that I’m sharing, the reference is 
right there.” It sort of does add, or kind of provide a baseline level of trust that this insight or 
thought has been memorialised in this book that someone chose to publish, and it lives on my 
bookshelf. There is some level of credibility that’s built by attaching your insider thoughts to 
that credible source. So, yeah, there’s definitely a tie between references, I guess, in citations 
to the physical setting of having a conversation and a book living on your bookshelf, that you 
point to. I think that’s an interesting connection beyond just existing as social objects that 
speak to your identity, as well. That’s another extension as well. I think that’s really 
interesting. 

Frode Hegland: Thanks for that. Bob. But afterward, Fabien, if you could elaborate on your 
comment in the chat, that would be really great. Bob, please. 

Video: https://youtu.be/i_dZmp59wGk?t=1460 

Bob Horn: Well, the first thing that comes to mind is: 

Have you looked at three-dimensional spaces on the internet? For example, Second Life, and 
what do you think about that? 

Jad Esber: Yeah. I mean, part of what people are proposing for the future of the internet is 
what I’m sure you guys have discussed in past sessions. Perhaps is like the metaverse, right? 
Which is essentially this idea of co-presence, and some level of physicality bridging the gap 


between being co-presented in a physical space, in a digital space. Second Life was a very 
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early example of some version of this. I haven’t spent too many iterations thinking about 
virtual spaces and whether they are apt at emulating the feeling of walking into a bookstore, 
or leafing through a bookshelf. But I think if you think about the sensory experience of being 
able to browse someone’s bookshelf, there are, obviously, parallels to the visual sensory 
experience. You can browse someone’s digital library. Perhaps there’s some level of tactile, 
you can pick up books, but it’s not really the same. But it’s missing a lot of the other sensory 
experiences, which provide a level of context. But certainly, allow for that serendipitous 
discovery that another doesn’t. Like the feed dynamic isn’t necessarily the most 
serendipitous. It’s it is to a degree, but it’s also very crafted. And it there isn’t really a level of 
play when you’re going around and looking at things that you do on a bookshelf, or in a 
bookstore. And so, Second Life does allow for that. Moving around, picking things up and 
exploring that you do in the physical world. So, I think it’s definitely bridging the gap to an 
extent, but missing a lot of the sensory experiences that we have in the physical world. I think 
we haven’t quite thought about how to bridge that gap. I know there are projects that are 
trying to make our experience of digital worlds more sensory, but I’m not quite sure how 
close we’ll get. So, that’s my initial thought, but feel free to jump in, by the way, I’d welcome 
other opinions and perspectives as well. 

Bob Horn: We’ve been discussing this a little bit, partially, at my initiative, and mostly at 
Frode’s urging us on. And I haven’t been in Second Life for, I don’t know, six, or seven, or 
eight years. But I have a friend who has, who’s there all the time, and says that there are 
people who have their personal libraries there. That there are university libraries. Their whole 
geographies, I’m told, of libraries. So, it may be an interesting angle, at some point. And if 
you do, I’d be interested, of course, in what you came up with. 

Jad Esber: Totally. Thank you for that pointer, yeah. There’s a multitude of projects right 
now that focus on extending Second Life, and kind of bringing in concepts around 
ownership, and physicality, and interoperability, so that the things that you own in Second 
Life, you can take with you, from that world, into others. Which, sort of, does bridge the gap 
between the physical world and the digital, because it doesn’t live within that siloed space, 
but actually is associated to you, and can be taken from one space to another. Very early in 
building that out, but that’s a big promise of Web3, so. There’re a lot of hands. So, I’ll pause 
there. 

Frode Hegland: Yeah, Fabien, if you could elaborate on what you were talking about, virtual 
bookshelf. 

Fabien Benetou: Yep. Well, actually it will be easier if P’ll share my screen. I don’t know if 
you can see. I have a Wiki that I’ve been maintaining for 10 plus years. And on top, you can 


see the visualisation of the edits when I started for this specific page. And these pages, as I 
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was saying in the chat, are sadly out of date, that’s been 10 years, actually, just for this page. 
But I was listing the different books I’ve read, with the date, what page I was. And if I take a 
random book, I have my notes, the (indistinct), and then the list of books that are related, let’s 
say, to the book. I don’t have it in VR or in 3D yet, but it’s definitely from that point wouldn’t 
be too hard, so... And I was thinking, I have personally a, kind of, (indistinct) that they’re 
hidden, but I have some books there and I have a white wall there and I love both because 
when I bring back if either I’m in someone else’s room or my own room. Usually, if ’'m in 
my own room, I’m excited by the book I’ve read or the one that I haven’t read yet. So it 
brings a lot of excitement. But also, if I have a goal in mind, a task at hand, let’s say, a 
presentation on Thursday, a thing that I haven’t finished yet, then it pulls me to something 
else. Whereas if I have the white wall it’s like a blank slate. And again, if I need to pull some 
references on books and whatnot. So, I always have that tension. And what usually happens 
is, when I go in a physical bookstore, or library, or bookshop, or friends, serendipity is 
indeed, it’s not the book I came here for, it’s the one next to it. Because I’m not able to make 
the link, and usually, if the creation has been done right, and arguably the algorithm, if it’s not 
actually computational, let’s say, if you use the doing annotation or any other basically 
annotation system, in order to sort the books or their references, then there should be some 
connection that were not obvious in the first place. So, to me, that’s the most, I'd say, exciting 
aspect of that. 

Jad Esber: This is amazing, by the way, Fabien. This is incredible that you’ve built this over 
a decade, that’s so cool. I think what’s also really interesting to extend on that thought, and 
just to kind of like, “yes” and that, there is a certain level of, I mean, I think what you’ve built 
is very utilitarian, but also the existence of the bookshelf as an expression of identity, I think 
is interesting. So, beyond just organising the books, and keeping them, storing them in a 
utilitarian way, then serving as signals of your identity, I think are really interesting. And so, I 
think a lot of platforms today cater to the utility. If you think about Pocket or even Goodreads 
to an extent, there is potentially an identity angle to Goodreads versus Tumblr, back in the 
day, or Myspace or (indistinct) which were much more identity-focused. So there is this 
distinction of utilitarian, organising, keeping things, annotating, etc. for yourself. But there’s 
also this identity element of like, by curating I am expressing my identity. And I think that’s 
also really interesting. 

Frode Hegland: Brandel, you’re next. But just wanted to highlight today to the new people 
in the room including you, Jad. This community, at the moment, is really leaning towards AR 
and VR. But in a couple of years’ time, what can happen? And that also includes projections 
and all kinds of different things, so we really are thinking connected with the physical, but 


also virtual on top. Brandel, please. 
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Brandel Zachernuk: So, I was really hooked on when you said that you like to be seen in 
that London bookstore. And it made me think about the fact that on Spotify, on YouTube, on 
Goodreads for the most part, we’re not seen at all, unless we’re on the specific, explicit page 
that is there for the purposes of representing us. So, YouTube does have a profile page. But 
nothing about the rest of our onward activity actually is represented within the context of 
that. If you compare that to being in the bookstore, you have your clothes on, you have your 
demeanour, and you can see the other participants. There’s a mutuality to being present in it, 
where you get to see that, rather than merely that a like button maybe is going up in real-time. 
And so, I’m wondering what kind of projective representation do you feel we need within the 
broader Web? Because even making a new curation page still silos that representation with an 
explicit place, and doesn’t give you the persistent reference that is your own physicality, and 
body wandering around the various places that you want to be at and be seeing at. Now, do 


you see that as something that there’s a solve to? Or how do you think about that? 
Jad Esber: Yeah, I think Bob alluded to this to a degree with Second Life. And the example 


of Second Life, I think the promise of co-presence in the digital world is really interesting, 
and potentially could solve for this, part of. I also go to cafes, not just because I like the 
coffee, because I like the aesthetic, and the opportunities to rub shoulders with other clientele 
that might be interesting, because this cafe is frequented by this sort of folk. And that doesn’t 
exist online as much. I mean, perhaps, if you’re going to a forum, and you frequent a specific 
subreddit, there is an element of like, “Oh, I’1l meet these types of folks or this chat group, 
and perhaps, I’Il be able to converse with these types of folks and be seen here.” But I think, 
how long you spend there, how you show up there, beyond just what you write. That all 
matters. And how you’re browsing, there’s a lot of elements that are really lost in current user 
interfaces. So, I think, yeah, Second Life-like spaces might solve for that, and allow us to 
present other parts of ourselves in these spaces, and measure time spent, and how we’re 
presenting, and what we’re bringing. But, yeah. I’m also fascinated by this idea of just 
existing in a space as a signal for who you are. And yeah, I also love that metaphor. And 
again, this is all stuff that I’m actively thinking about and would love sort of any additional 
insights, if anyone has thoughts on this, please do share, as well. This is, by no means, just a 
monologue from my direction. 

Frode Hegland: Oh, I think you’re going to get a lot of perspectives. and I will move into... 
We’re very lucky to have Dene here, who’s been working with electronic literature. I will let 
her speak for herself, but what they’re doing is just phenomenally important work. 

Dene Grigar: Thank you. That’s a nice introduction. I am the managing director, one of the 
founders, and the curator of The NEXT. And The NEXT is a virtual museum, slash library, 


slash preservation space that contains, right now, 34 collections of about 3,000 works of 
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born-digital art and expressive writing. What we generally call ‘electronic literature’. But I’ve 
unpacked that word a little bit for you. And I think this corresponds to a little bit of what 
you’e talking about in that when we cut when I collect when I curate work I’m not picking 
particular works to go in The NEXT, I’m taking full collections. So, artists turn over their 
entire collections to us, and then that becomes part of The NEXT collections. So it’s been 
interesting watching what artists collect. So it’s not just their own works, it’s the works of 
other artists. And the interesting, historical, cultural aspect of it is to see, in particular time 
frames, artists before the advent of the browser, for example, what they collected, and who 
they were collecting. Michael Joyce, Stuart Moulthrop, Voyager, stuff like that. Then the 
Web, the browser, and the net art period, and the rise of Flash, looking to see that I have five 
copies of Firefly by Nina Larson because people were collecting that work. Jason Nelson’s 
work. A lot of his games are very popular. So it’s been interesting to watch this kind of 
triangulation of what becomes popular, and then the search engine that we built pulls that up. 
It lets you see that, “Oh, there’s five copies of this. There’s three copies of that. Oh, there’s 
seven versions of Michael Joyce’s afternoon, a story.” To see what’s been so important that 
there’s even been updates, so that it stays alive over the course of 30 years. One other thing 
I'll mention, back to your early comment, I have a whole print book library in my house, 
despite the fact I was in a flood in 1975 and lost everything I owned, I rebuilt my library and 
I have something like 5,000 volumes of books, I collect books. But it’s always interesting for 
me, to have guests at my house and they never look at my bookshelf. And the first thing I do 
when I go to someone’s house, I see books is like, “Oh, what are you reading? What do you 
collect?” And so, looking at having The NEXT and all that 3,000 works of art and then my 
bookshelf, and realising that people really aren’t looking and thinking about what this means. 
The identity for the field, my own personal taste, I call it my own personal taste, which is 
very diverse. So, I think there’s a lot to be said about people’s interest in this. And I think it’s 
that kind of intellectual laziness that drives people to just allow themselves to be swept away 
by algorithms, and not intervene on their own and take ownership over what they’ re 
consuming. And I'll leave it at that. Thank you. 

Jad Esber: Yeah, I love that. Thank you for sharing. And that’s a fascinating project, as well. 
I'd love to dig in further. I think you bring up a really good point around shared interests 
being really key and connecting the right type of folks, who are interested in exploring each 
others libraries. Because not everyone that comes into my house is interested in the books 
that I’m reading, because, perhaps, they’re from a different field, they’re just not as curious 
about the same fields. But there is a huge amount of people that potentially are. I mean, 
within this group, we’re all interested in similar things. And we found each other through the 


internet. And so, there is this element of, what if the people walking into your library, Dene, 
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are also folks that share the same interests as you? That would actively look and browse 
through your library and are deeply interested in the topics that you’re interested in so there is 
something to be said around how can we make sure that people that are interested in the same 
things are walking into each others’ spaces? And the interest-based graphs exist on the Web. 
Thinking about who is interested in what, and how can we go into each others’ spaces. And 
browse, or collecting, or curating, or creating is a part of what many algorithms try to do, for 
better or for worse. But sometimes leave us in echo chambers, right? And we’re in one 
neighbourhood and can’t leave, and that’s part of the problem. But yeah, there is something to 
be said about that. And I think just to go back to the earlier comment that the Dene made 
around the inspirations behind artists’ work. I would love to be able to explore what inspired 
my favourite artist’s music, and what went into it and go back and listen to that. And I think, 
part of again, Web3’s promise is this idea of provenance, seeing how things have evolved and 
how they’ve become. And crediting everyone in that lineage. So, if I borrowed from Dene’s 
work, and I built on it, and that was part of what inspired me, then she should get some credit. 
And that idea of provenance, and lineage, and giving credit back, and building incentive 
systems that allow people to build works that will inspire others to continue to build on top of 
my work is a really interesting proposal for the future of the internet. And so, I just wanted to 
share that as well. 

Frode Hegland: That’s great. Anything back from you, Dene, on that? Before we move to 
Mark? 

Dene Grigar: Well, I think provenance is really important. And what I do in my own lab is to 
establish provenance. Even if you go to The NEXT and you look at the works, it’Il say where 
we got the work from, who gave it to us, the date they gave it to us, and if there’s some other 
story that goes with it. For example, I just received a donation from a woman whose daughter 
went to Brown University and studied under Coover, Robert Coover. And she gave me a copy 
of some of the early hypertext works, and one was Michael Joyce’s Afternoon Story and it 
was signed. The little floppy disk was signed, on the label, by Michael and she said, “I didn’t 
notice there was a signature. I don’t know why there’d be a signature on it.” And, of course, 
the answer is, if you know anything about the history is that Joyce and Coover were friends, 
there’s this whole line of this relationship and Coover was the first to review Michael Joyce, 
and made him famous in the New York times, in 1992. So, I told her that story, and she’s like, 
Oh, my god. I didn’t know that.” So, just having that story for future generations to 
understand the relationships, and how ideas and taste evolve over time, and who were the 
movers and shakers behind some of that interest, so. Thank you. 

Bob Horn: Could I ask, just briefly, what the name of your site is or something? Because it 


went by so fast that I couldn’t even write it down. 
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Dene Grigar: https://the-next.eliterature.org/. Yeah, and I encouraged you to look at it. 


Frode Hegland: Dene, this is really grist for the mill of a lot of what we’re talking about 
here. Because, with Jad’s notions of identity sharing via the media we consume, and a lot of 
the visualisations we’re looking at in VR. One of the things we’ve talked about over the last 
few weeks is guided tours of work where you could see the hands of the author or somebody 
pointing out things whether it’s a mural, or a book, or whatever. And then, to be able to find a 
way to have the meta-information you just talked about, be able to enter the room, maybe it 
could be simply recorded as you saying it, and that is tagged to be attached to these works. 
Many wonderful layers, I could go on forever. And I expect mark will follow up. 

Mark Anderson: Hi. I just think, they’re really reflections, more than anything else. Because 
one of the things that really brought me up was this idea of books being a performative thing, 
which I still can’t get my head around. It’s not something I’ve encountered, and I don’t see it 
reflected in the world in which I live. So maybe a generational drift in things. For instance, 
behind me you might guess, I suppose, I’m a programmer. Actually what that shows is it’s me 
trying to understand how things work, and I need them that close to my computer. My library 
is scattered across the house, mainly to distribute weight through a rather old crumbly 
Victorian house. So, I have to be careful where we put the bookcases. I’m just, really 
reflecting how totally alien I find the notion of books, I certainly don’t have... I struggled to 
think of, I never placed a book with the intention it’Il be seen in that position by somebody 
else. And this is sort of not a pushback, it’s just my reflection on what I’m hearing. Because I 
find it very interesting because it had never occurred to me. I never, ever thought of it in 
those terms. The other sad thing about that means that, so, are the books merely 
performative? Or the content is there? I mean, one of the interesting thing I’ve been trying to 
do in this group is trying to find ways just to share the list of the books that are on my shelf. 
Not because they are any reflection of myself, but literally, I actually have some books that 
are quite hard to find, and people might want to know that it was possible to find a copy. And 
whether they need to come and physically see it, or we could scan something. The point is, 
“No, I have these. This is a place you can find this book.” And it’s interesting that that’s 
actually really hard to do. Most systems don’t help because, I mean, the tragedy of 
recommender systems is they make us so inward-looking. So, instead of actually rewarding 
our curiosity, or making us look across our divides, they basically say, “Right. You lot are a 
bunch. You go stand over there.” Job done, and (the) recommender system moves on to 
categorising the next thing. So, if I try to read outside my normal purview, and I’m constantly 
reflecting on the fact that the recommended system is one step behind saying, “Oh, right. 
You’re now interested in...” No, I’m not. I’m trying to learn a bit about it. But certainly, this 


is not my area of interest in the sense that I now want to be amidst lots of people who like 
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this. I’m interested in people who are interested by it, but I think those are two very different 
things. So, I don’t know the answers, but I just raise those, I suppose, as provocations. 
Because that’s something that, at the moment, our systems are really bad at allowing us to 
share content other than as a sort of humblebrag. Or, in your beautifully curated life on 
Pinterest, or whatever. Anyway, I’ Il stop there. 

Jad Esber: Yeah, thank you for sharing that. I think it does exist on a spectrum, the identity 
expressive versus utilitarian need that it solves. But if you take the example of clothing, that 
might help it a little bit more. So, if we’re wearing a t-shirt, perhaps there’s a utilitarian need, 
but there is also a performative, or identity expressive need that it solves the way we dress, 
speaks to who we are as well. So I think the notion of a social object being identity 
expressive, I think is what I was trying to convey. Think, if you think about magazines on a 
coffee table. Or you think about the art books that live scattered around your living room, 
perhaps. That is trying to signal something about yourself. The magazines we read as well. If 
I’m reading Vogue, I’m trying to say something about who I am, and what I’m interested in 
reading. The Times, or The Guardian, or another newspaper is also very identity expressive. 
And taking it out on the train and making sure people see what I’m reading is also identity 
expressive. So, I think that everything sort of around what we consume and what we wear 
and what we identify with being a signal of who we are. It’s what I was trying to convey 
there. But I think you make a very good point. The books next to your computer are there 
because they’re within reach. You’re writing a paper about something and it’s right there. And 
so, there is a utilitarian need for the way you organise your bookshelf. The way you organise 
your bookshelf can be identity expressive or utilitarian. I'll give you another example. On my 
bookshelf, I have a few books that are turned face forward, and a few that I don’t really want 
people to see them, because I’m not really that proud of them. And I have a book that’s 
signed by the author, I’ll make sure it’s really easy for people to open it and see the signature. 
And so, there is an identity expressive element to the way I organise my bookshelves as well 
that’s not just utilitarian. So, I think another point to illustrate that angle. 

Mark Anderson: To pull us back to our, and as a sub-focus on AR, VR, it just occurred to 
me it’s something that, the (indistinct) reminder that Dene was talking about, people don’t 
look at the bookshelves. I’m thinking, yeah and certainly not saying I miss, and it happens 
less frequently that the evening ends up with a dinner table just loaded with piles of books 
that have been retrieved from all over the house and are actually part of the conversation 
that’s going on. And one thing that some of our new tools would be nice to help us recreate 
that, especially maybe, if we’re not meeting in the same physical space, is to have that 
element of recall of these artefacts, or at least some of the pertinent parts of the content 


they’re in. It would be really useful to have because the fact that you bothered to walk up two 
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flights of stairs or something to go and get some book off the top shelf, because that’s, in a 
sense, part of the conversation going on, I think is quite interesting and something we’ ve sort 
of lost anyway. Ill let it carry on. 

Frode Hegland: It’s interesting to hear what you say there, Mark, because in the calls we 
have, you’re the one who most often will, “Look, the book arrived. Look, I have this copy 
now.” And then we all get really annoyed at you because we have to buy the same damn 
book. So, I think we’re talking about different ways and to different audiences, not 
necessarily to dinner guests. But for your community of this thing, you’re very happy to 
share. Which is interesting it’s also two points, to use my hand in the air here. One of them is, 
clothing came up as well. And some kind of study, I read showed that, we don’t buy clothing 
we like, we buy clothing that is the kind of clothing we expect people like us to buy. So, even 
somebody who is really, “I don’t care about fashion” is making a very strong fashion 
statement. They’re saying they don’t care. Which is anti-snobbery, maybe. You could say that 
I’m wondering how that enters into this. But also, when we talk about curation, it’s so 
fascinating how, in this discussion, music and books are almost interchangeable from this 
particular aspect. And what I found is, I don’t subscribe to Spotify, I never have, because I 
didn’t like the way the songs were mixed. But what I do really like, and I find amazing, is 
YouTube mixes. I pay for YouTube premium so I don’t have the ads. That means Ill have an 
hour, an hour and a half, maybe two-hour mixes by DJs who really represent my taste. Which 
is a fantastic new thing. We didn’t have that opportunity before. So that is a few people. And 
there, the YouTube algorithm tends to put me in direction of something similar. But also this 
is for music when I work. It’s not for finding new interesting Jazz. When I play this music, 
when I’m out driving with my family, I hear how incredibly inane and boring it is. It is 
designed for backgrounding. So the question then becomes, maybe, do we want to have 
different shelves? Different bookshelves for different aspects of our lives? And then we’re 
moving back into the virtuality of it all. That was my hand up. Mark, is your hand up for a 
new point? Okay, Fabien? 

Fabien Benetou: Yeah a couple of points. The first to me, the dearest to me, let’s say, is the 
provenance aspect. I’m really pissed or annoyed when people don’t cite sources. I would 
have a normal conversation about a recipe or anything completely casual, doesn’t have to be 
academic, and if that person didn’t invent it themselves, I’m annoyed if there is not some way 
for me to look back to where it came from. And I think, honestly, a lot of the energy we waste 
as a species comes from that. If you’re not aware, of course, of the source, you can’t cite it. 
But if you learn it from somewhere not doing that work, I think is really detrimental. Because 
we don’t have to have the same thought twice if we don’t want to. And if we just have it 


again, it’s just such a waste of resources. And especially since I’m not a physician, and I don’t 
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specialise in memory, but from what I understood, source memory is the type of memory 
where you recall, not the information, but where you got it from. And apparently, it’s one of 
the most demanding. So for example, you learn about, let’s say, a book, and you know 
somebody told you about that book, and that’s going to be much harder but eventually, if you 
don’t remember the book itself, but the person who told you about it, you can find it back. 
So, basically, if as a species, we have such a hard time providing sources and understanding 
where something comes from, I think it’s really terrible. It does piss me off, to be honest. And 
I don’t know if metadata, in general, is an answer. If having some properly formatted, any 
kind of representation of it, ’m not going to remember the ISBN of the book, on the top of 
my head in a conversation, but I’m wondering in terms of, let’s say if blockchain can solve 
that? Can Web3 solve it? Especially you mentioned the, let’s say, a chain of value. If you 
have a source or the reference of somewhere else whose work you’re using, it is fair to 
reattribute it back. They were part of how you came to produce something new. So, I’m quite 
curious about where this is going to be. 

Jad Esber: Yes, thank you for that question. And, yeah. I think there are a few points. First 
is, I’m going to just comment really quickly on this idea of provenance. And I want to just 
jump back to answer some of Frode’s comments, as well. But I think, one thing that you 
highlighted, Fabien, is how hard it is for us to remember where we learned something or got 
something. And part of the problem is that, so much of citing and sourcing is so proactive and 
requires human effort. And if things were designed where it was just built into the process. 
One of the projects I worked on at YouTube was a way for creators to take existing videos 
and build on them. So, remixing essentially. And in the process of creating content, I’d have 
to take a snippet and build on it. And that is built into the creation process. The provenance, 
the citing are very natural to how I’m creating content. TikTok is really good at this too. And 
so I wonder if there are, again, design systems that allow us to build in provenance and make 
it really user-friendly and intuitive to remove the friction around having to remember the 
source and cite. We’re lazy creatures. We want that to be part of our flow. TikTok duets 
feature and stitching is brilliant. It builds in provenance into the flow. And so, that’s just one 
thought. In terms of how blockchains help. So, part of what is a blockchain other than a 
public record of who owns what, and how things are being transacted. If there was a way if 
we go back to TikTok stitching, or YouTube quoting a specific part of a video, and building 
on it, if that chain of events was tracked and publicly accessible, and there was a way for me 
to pass value down that chain to everyone that contributed to this new creative work, that that 
would be really cool. And that’s part of the promise. This idea of keeping track of how 
everything is moving, and being able to then distribute value in an automated way. So, that’s 
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perhaps tying in with some of what we talked about with Mark, around identity expression. I 
think this all comes back to the human need to be heard, and understood, and seen, and there 
are phases in our life, where we’re figuring out who we are, and we don’t really have our 
identities figured out yet. So, if you think about a lot of teenagers, they will have posters on 
their walls to express what they’re consuming or who they’re interested in. And they are 
figuring out who they are. And part of them figuring out who they are is talking about what 
they’re consuming, and through what they’re consuming, they’re figuring out their identities. 
I grew up writing poetry on the internet because I was trying to express my experiences, and 
figure out who I was. And so, I think what I’m trying to say is that there will be periods of 
our life where the need to be seen, heard, understood or we’re figuring out, and forming our 
identities are a bigger need. And so, the identity expressive element of para-socially 
expressing or consuming plays a bigger part. And then, perhaps when we’re more settled with 
our identity, and we’re not really looking to perform that, becomes more of a background 
thing. Although, it doesn’t completely disappear because we are always looking to be heard, 
seen, and understood. That’s very human. So, I’ll pause there. I can keep going, but I’ll pause 


because I see there are a few other hands. 


Frode Hegland: Yeah, I’ll give the torch to Dave Millard. But just on that identity, I have a 
four-and-a-half-year-old boy, Edgar, who is wonderful. And he currently likes sword fighting 
and the colour pink. He is very feminine, very masculine, very mixed up, as he should be. So, 
it’s interesting, from a parental, rather than from just an old man’s perspective to think about 
the shaping of identity, and putting our posters and so on. It’s so easy to think about life from 
the point we are in life, and you’re pointing to a teenage part, which none of us are in. So, I 
really appreciate that being brought into the conversation. Mr. Millard? 

David Millard: Yeah, thanks, Frode. Hi, everyone. Sorry, I joined a few minutes late, so I 
missed the introductions at the beginning. But, yeah. Thank you. It’s a really interesting talk. 
One of the things we haven’t talked about is kind of the opposite of performative expression, 
which is privacy. One of the things, a bit like Mark, I’ve kind of learned about myself 
listening to everyone’s talking about this, is how deeply introverted I am, and how I really 
don’t want to let anybody know about me, thank you very much, unless I really want them to. 
This might be because I teach social network and media analytics to our computer scientists. 
So, one of the things I teach them about is inference, for example, profiling, I’m reminded of 
the very early Facebook studies done in the 2000s, about the predictive power of keywords. 
So, you’d express your interests through a series of keywords. And those researchers were 
able to achieve 90% accuracy on things like sexuality. This is an American study, so 
republican, democratic preferences. Afro-American, Caucasian, these kinds of things. So I do 


wonder whether or not there’s a whole element to this, which is subversive or exists in that 
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commercial realm that we ought to think about. I’m also struck about that last comment, 
actually, that you mentioned, which was about people finding their identities. Because I’ve 
also been involved in some research looking at how kids use social media. And one of the 
interesting things about the way that children use social media, including some children that 
shouldn’t be using social media, because they’re pretty 13 or whatever the cut-off date is. Is 
that they don’t use it in a very sophisticated way. And we were trying to find out why that 
was because we all have this impression of children as being naturally able. There’s the myth 
of the digital native and all that kind of stuff. And it’s precisely because of this identity 
construction. That was one of the things that came out in our research. So, kids won’t expose 
themselves to the network, because they’re worried about their self-presentation. They’re 
much more self-conscious than adults are. So they invest in dyadic relationships. Close 
friendships, direct messaging, rather than broadcasting identity. So I think there’s an opposite 
side to this. And it may well be that, for some people, this performative aspect is particularly 
important. But for other people, this performative aspect is actually quite frightening, or off- 
putting, or just not very natural. And I just thought I wanted to throw that into the mix. I 
thought it was an interesting counter observation. 

Jad Esber: Absolutely. Thank you for sharing that. To reflect on my experience growing up 
writing online. I wrote poetry, not because I wanted other people to read, it was actually very 
much for myself. And I did it anonymously. I wasn’t looking for any kind of building of 
credibility or anything like that. It was for me a form of healing. It was for me a form of just 
figuring out who I was. But if someone did read my poetry, and it did resonate with them, and 
they did connect with me, then I welcomed that. So, it wasn’t necessarily a performative 
thing. But it was a way for me to do something for myself that, if it connected with someone 
else, that was welcomed. I think to go back to the physical metaphor of a bookshelf. Part of 
my bookshelf will have books that I’ll present, and have upfront and want everyone to see, 
but I also have a book box with trinkets that are out of sight and are just for me. And that 
perhaps there are people who will come into my space and I’1l show them what’s in that box, 
selectively. And I'll pull them out, and kind of walk them through the trinkets. And then, I'll 
have some that are private, and are not for anyone else. So, I totally agree. If we think about 
digital spaces, if we were to emulate a bookshelf online, there will be elements, perhaps, that 
I would want to present to the world outwardly. There are elements that are for myself. There 
are elements that I want to present in a selective manner. And I think back to Frode’s point 
around bookshelves for various parts of my identity. I think that’s really important. There 
might be some that I will want to publicly present, and others that I won’t. If you think about 
a lot of social platforms, how young people use social platforms, think about Instagram. 
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And that’s because they had accounts that they used for performative reasons. And they had 
accounts that they used for themselves. And had accounts for specific parts of their identity. 
And that’s because we’re solving different needs through this idea of para-socially curating 
and putting out there what we’re interested in. So, just riffing on your point. Not necessarily 
addressing it, but sort of adding colour to it. 

David Millard: No, that’s great. Thank you. So, you’re right about the multiple accounts 
thing. I had a student, a few years ago, who’s looking at privacy protection strategies. I’m 
basically saying, people, don’t necessarily use the preferences on their social media 
platforms, who can see my stuff. They actually engage differently with those platforms. So 
they do like that, as you said. They have different platforms, or they have different accounts, 
for different audiences. They use loads of fascinating stuff, things like social stenography, 
which is, if they have in-jokes or hidden messages to certain crowds, that they will put in 
them, their feeds will never miss it. There are all of these really subtle means that people use. 
I’m sure that all comes into play for this kind of stuff as well. 

Jad Esber: Totally. Ill add to that really quickly. So, if you look at... I did a study of Twitter 
bios, and it’s really interesting to look at how, as you said, young folks will put very cryptic 
acronyms that indicate or signal their fanships. They’re looking for other folks who are 
interested in the same K-pop band, for example. And that acronym in the bio will be a signal 
to that audience. Like, come follow me, connect with me around this topic, just because the 
acronym is in there. A lot of queer folks will also have very subtle things in their bios, on 
their profile to indicate that. But only other queer folks will be aware of. And so, again, it’s 
not something you necessarily want to be super public and performative about, but for the 
right folk, you want them to see and connect with. So, yeah. Super interesting how folks have 
designed their own way of using these things to solve for very specific needs. 

Frode Hegland: Just before I let you go, Dave. Did you say steganography or did you say 
stenography? 

David Millard: | think it’s steganography. It’s normally referred to as hiding data inside 
other data but in a social context. It was exactly what Jad and I was just saying about using 
different hashtags or just references, quotes that only certain groups would recognise that 
kind of stuff, even if they’re from Hamilton. 

Frode Hegland: Brendan, I see you’re ready to pounce here. But just really briefly, one of 
the things I did for my PhD thesis is, study the history of citations and references. And 
they’re not that old. And they’re based around this, kind of, let’s call it, “anal notion” we have 
today that thing should be in the correct box, in the correct order, if it isn’t, it doesn’t belong 
in the correct academic discipline. Earlier this morning, Dave, Mark, and I were discussing 
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have. It’s crazy stuff. But before we got into that, we have a profession, therefore, we need a 
code of how to do it. The way people cited each other, of course, was exactly like this. The 
more obscure the better, because then you would really know that your readers understood 
the same space. So it’ s interesting to see how that is sliding along, on a similar parallel line. 
Brendan, please. Unless Jad has something specific on that point. 

Jad Esber: I was just sourcing a Twitter bio to show you guys. So, maybe, if I find one, I'll 
walk through it and show you how various acronyms are indicating various things. And I was 
just trying to pull it from a paper that I wrote. But, yeah. Sorry, go ahead, Brendan. 

Frode Hegland: Okay, yeah. When you’re ready, please put that in. Brendan? 

Brendan Langen: Cool. Jad, really neat to hear you talk through, just really everything 
around identity as a scene online. It’s a point of a lot of the research I’m doing as well. So, 
interesting overlaps. First, I'll kind of make a comment, and then I have a question for you 
that’s a little off base of what we talked about. But the bookshelf, as a representation, is 
extremely neat to think about when you have a human in the loop because that’s really where 
contextual recommendations actually come to life. This idea of an algorithm saying that 
we’ve read 70% of the same books, and I have not read this one text that you have held really 
near and dear to you might be helpful but, in all honesty, that’s going to fall short of you 
being able to share detail on why this might be interesting to me. So I guess to, kind of, pivot 
into a question, one of my favourite things that I read last year was something you did with, I 
forget the fella’s name, Scott, around reputation systems and novel approach, and so, I’m 
studying a little bit in this Web3 area, and the idea of splitting reputation, and economic value 
is really neat. And I'd love to hear you talk a little bit more about ‘Koodos’ and how, either 
you’re integrating that, or what experiments you’re trying to run in order to bring like 
curation and reputation into the fold. I guess like, what kind of experiments are you working 
on with ‘Koodos’ around this reputational aspect? 

Jad Esber: Yeah, absolutely. I’m happy to share more. But before I do that, I actually found 
an example of a Twitter bio, I'll really quickly share, and then, I’m happy to answer that 
question, Brendan. So this is from a thing I put together a while ago, and if we look at the 
username here. So, ‘katie, exclamation mark, seven, four Dune’. So, the seven here actually 
is supposed to signal to all BTS fans, BTS being a K-pop band that she is part of that group, 
that fan community. It’s just that simple seven next to her name. Four Dune is basically a way 
for her to indicate that she is a very big fan of Dune, the movie, and Timothée Chalamet, the 
actor. And pinned at the top of her Twitter account is this list of the bands or the communities 
that she stands, stands meaning, being a big fan of. And so, again, sort of like, very 
cryptically announcing the fan communities she’s a part of just in her name, but also, very 


actively pinning the rest of the fan communities that she’s a member of, or a part of. But, 
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yeah. I just want to share that really quickly. So, to address, Brendan, your questions, just for 
folks who aren’t aware of the piece, it’s basically a paper that I wrote about how to decouple 
reputation from financial gain in system and reputation systems, where there might be a 
token. So, a lot of Web3 projects promise community contributions will earn you money. And 
the response that myself and Scott Kominers wrote was around, “Hey, it doesn’t actually 
make sense for intrinsic motivational reasons, for contributions to earn you money. In fact, if 
you’re trying to build a reputation system, you should develop a system to gain reputation, 
that perhaps spins off some form of financial gain. { So, that’s, sort of, the paper. And I can 
link it in in the chat, as well, for folks who are interested. So, a lot of what I think about with 
‘Koodos’, the company that I’m working on, is this idea of, how can people build these 
digital spaces that represent who they are, and how can that may remain a safe space for 
identity expression, and perhaps, even solving some of the utilitarian needs. But then, how 
can we also enable folks, or enable the system, to curate at large, source from across these 
various spaces that people are building, to surface things that are interesting in ways that 
aren’t necessarily super algorithmic. And so, a lot of what we think about the experiments we 
run around how can we enable people to build reputation around what it is that they are 
curating in their spaces. So, does Mark’s curation of books in his bookshelf give him some 
level of reputation in specific fields? That then allows us to point to him as a potential expert 
on that space. Those are a lot of the experiments that we’re interested in running, just sort of, 
very high level without getting too in the weeds. But I’m happy to discuss, if you’re really 
interested in the weeds of all of that, without boring everyone, I’m happy to take that 


conversation as well. 

Brendan Langen: Yeah. I'll reach out to you because I’m following the weeds there. 
Jad Esber: Yeah, for sure. 

Brendan Langen: Thanks for the high-level answer. 

Jad Esber: No worries, of course. 


Frode Hegland: Jad, I just wanted to say, after Bob and Fabien now, I would really 
appreciate it if you go into sales mode, and really pitch what you’re working on. I think, if we 
honestly say, it’s sales mode, it becomes a lot easier. We all have passions, there’s nothing 
wrong with being pushy in the right environment, and this is definitely the right environment. 
Bob? 

Bob Horn: Well, I noticed that your slides are quite visual and that you just mentioned 
visual. I wonder if, in your poetry life, you’ve thought about broadsheets? And whether you 
would have broadsheets in the background of coming to a presentation like this, for example, 


so that you could turn around and point to one and say, “Oh, look at this.” 


Jad Esber: I’m not sure if the question is if I... I’m sorry, what was the question specifically 


263 


about? 

Bob Horn: Well, I noticed you mentioned that you are a poet, and poets often, at least in 
times gone by, printed their poems on larger broadsheets that were visual. And I associated 
that with, maybe, in addition to bookshelves, you might have those on a wall, in some sort of 
way, and wondered if you’d thought about it, and would do it, and would show us. 

Jad Esber: Yeah. So, the poetry that I used to write growing up was very visual, and it used 
metaphors of nature to express feelings and emotions. So, it’s visual in that sense. But I am, 
by no means, a visual artist or not visual in that sense. So, I haven’t explored using or pairing 
my poetry with visual compliments. Although, that sounds very interesting. So, I haven’t 
explored that. Most of my poetry is visual in the language that I use. And the visuals that 
come up in people’s minds. I tend to really love metaphors. Although, I realise that 
sometimes they can be confining, as well. Because we’re so limited to just that metaphor. 
And if I were to give you an example of one metaphor, or one word that I really dislike in 
the Web3 world it’s the ‘wallet’. I’m not sure how familiar you are with the metaphor of a 
wallet in Web3, but it’s very focused on coins and financial things, like what live in your 
physical wallet, whereas what a lot of wallets are today are containers for identity and not just 
the financial things you hold. You might say, ‘Well, actually, if you look into my wallet, I 
have pictures of my kids and my dog or whatever.’ And so, there is some level of storing 
some social objects that express my identity. I share that just to say that the words we use, 
and the metaphors that we use, do end up also constraining us because a lot of the projects 
that are coming out of the space are so focused on the wallet metaphor. So, that was a very 
roundabout answer to say that I haven’t explored broadsheets, and I don’t have anything 
visual to share with my poetry right now. 

Bob Horn: What is, just maybe, in a sentence or two, what is Web3? 

Jad Esber: Okay, yeah. Sure. So, Web3, in a very short sense, is what comes after Web2, 
where Web2 is what we as, sort of, the last phase of the internet that relied on reading and 
writing content. So if you think about Web! being read-only, and Web2 being read and write, 
where we can publish as well. Web3 is read-write and on. So, there is an element of 
ownership for what we produce on the internet. And so, that’s, in short, what Web3 is. A lot 
of people associate Web3 with blockchains, because they are the technology that allows us to 
track ownership. So that’s what Web3 is in a very brief explanation. Brendan, as someone 
who’s deep in this space, feel free to add as well to that, if I’ve missed anything. 


Bob Horn: Thank you. 


Brendan Langen: I guess the one piece that is interesting in the wallet metaphor is that, I 
guess, the Web2 metaphor for identity sharing was like a profile. And I guess I would love to 


hear your opinion on comparing those two and the limitations of what even a profile provides 
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as a metaphor. Because there are holes in identity if you’re just a profile. 

Jad Esber: Totally, yeah. Again, what is a profile, right? It’s a very two-dimensional, like... 
What was a profile before we had Facebook profiles? A profile when you publish something 
is a little bit of text about you, perhaps it’s a profile picture, just a little bit about you. But 
what they’ve become is, they are containers for photos that we produce and there are spaces 
for us to share our interests and we’re creating a bunch of stuff that’s a part of that profile. 
And so, again, the limiting aspect of the term ‘profile’ exists a lot of on what’s been 
developed today, again, just hinges on the fact that it’s tied to a username and a profile picture 
and a little bio. It’s very limiting. I think that’s another really good example. Using the term 
‘wallet’ today, again, is limiting us in a similar way to how profiles limited us in Web2. If we 
were to think about wallets as the new profile. So that’s a really good point I actually hadn’t 
made that connection, so thank you. 

Fabien Benetou: Thank you. Honestly, I hope there’s going to be, let’s say, a bridge to the 
pitch. But to be a little bit provocative, honestly, when I hear Web3, I’m not very excited. 
Because I’ve been burnt before. I checked bitcoin in 2010 or something like this, and 
Ethereum, and all that. And honestly, I love the promise of the Cypherpunk movement or the 
ideology behind it. And to be actually decentralised or to challenge the financial system and 
its abuse speaks to me. I get behind that. But then, when I see the concentration back behind 
the different blockchains, most of the blockchains are rougher, then I’m like, “Well, we made 
the dream”. Again, from my understanding of the finance behind all this. And yet, I have 
tension, because I want to get excited, like I said, the dream should still live. As I was briefly 
mentioned in the chat earlier, civilians, capitalism, and the difference between doing 
something in public, and doing something on Facebook, it’s not the same. First, because it’s 
not in public, it’s not a proper platform. But then, even if you do it publicly on Facebook, is 
the system to issue value and transform that to money. And I’m very naive, I’m not an 
economist, but I think people should pay for stuff. It’s easy. I mean, it’s simple, at least. So, if 
I love your poetry, and I can find a way that can help you, then I pay for it. There is no need 
for an intermediary, in between, especially if it’s at the cost of privacy and potentially 
democracy behind. So that’s my tension, I want to find a way. That’s why I’m also about 
provenance, and how we have a chain of sources, and we can reattribute people back down 
the line. Again, I love that. But when I hear Web3 I’m like, “Do we need this?” Or can we 
can, for example, and I don’t like Visa or Mastercard, but I’m wondering if relying on the 
centralised payment system is still less worse than a Cypherpunk dream that’s been hijacked. 
Brendan Langen: Yeah, I mean, I share your exact perspective. I think Web3 has been 
tainted by the hyper-financialisation that we’ve seen. And that’s why, when Bob asked what 


is Web3, it’s just what’s after Web2. I don’t necessarily tie it, from my perspective to crypto 
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necessarily. I think that is a means to that end but isn’t necessarily the only option. There are 
many other ways that people are exploring, that serve some of the similar outcomes that we 
want to see. And so, I agree with you. I think right now, the version of Web3 that we’re 
seeing is horrible, crypto art and buying and selling of NFTs as stock units is definitely not 
the vision of the internet that we want. And I think it’s a very skeuomorphic early version of 
it that will fade away and it’s starting to. But I think the vision that a lot of the more enduring 
projects in the space have around provenance and ownership, do exist. There are projects that 
exist that are thinking about things in that way. And so, we’re in the very early stages of 
people looking for a quick buck, because there’s a lot of money to be made in the space, and 
that will all die out, and the enduring projects will last. And so, I think decoupling Web3 from 
blockchain, like Web3 is what is after Web2, and blockchain is one of the technologies that 
we can be building on top of, is how I look at it. And stripping away the hyper- 
financialisation, skeuomorphic approaches that we’re seeing right now from all of that. And 
then, recognising also, that the term Web3 has a lot of weight because it’s used in the space to 
describe a lot of these really silly projects and scams that we’re seeing today. So, I see why 


there is tension around the use of that term. 


Frode Hegland: One of the discussions I had with the upcoming Future of Text work, I’m 
embarrassed right now, I can’t remember exactly who it was (Dave Croker), but the point 
was made that, version numbers aren’t very useful. This was in reference to Visual-Meta, but 
I think it relates to Web2. Because if the change is small you don’t really need a new version 
number, and if it’s big enough it’s obvious. So, I think this Web3, I think we all kind of agree 
here, is basically marketing. 

Jad Esber: It’s just a term, yeah. I think it’s just a term that people are using to describe the 
next iteration of the Web. And again, as I said, words have a lot of weight and I’m sure 
everyone here agrees that words matter. So yeah, I think, when I reference it, usually I’m 
pointing to this idea of read-write-own. And own being a new entry in the Web. So, yeah. 
Bob Horn: I was wondering whether it was going to refer to the Semantic Web, which Tim 
Berners-Lee was promoting some years ago. Although, not with a number. But I thought 
maybe they’ve added a number three to it. But I’m waiting for the Semantic Web, as well. 
Jad Esber: Totally. I think the Semantic Web has inspired a lot of people who are interested 
in Web3. So, I think there is a returning back to the origins of the internet, right? Ted 
Nelson’s thinking as well as a big inspiration behind a lot of current thinking in this space. 
It’s very interesting to see us loop back almost to the original vision of the Web. Yeah, 
totally. 

Brandel Zachernuk: You talked a little bit about algorithms, and the way that algorithms 


select. And painted it as ineffable or inaccessible. But the reality of algorithms is that they’re 
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just the policy decisions of a given governing organisation. And based on the data they have, 
they can make different decisions. They can present and promote different algorithms. And so 
that ‘Forgotify’ is a take on upending the predominant deciding algorithm and giving 
somebody the ability through some measure of the same data, to make a different set of 
decisions about what to be recommended. The idea that I didn’t get fully baked, that I was 
thinking about is the way that a bookshelf is an algorithm itself, as well. It’s a set of decisions 
or policies about what to put on it. And you can have a bookshelf, which is the result of 
explicit, concrete decisions like that. You can have a meta bookshelf, which is the set of 
decisions that put things on it, that causes you to decide it. And just thinking about the way 
that there is this continuum between the unreachable algorithms that people, like YouTube, 
like Spotify, put out, and the kinds of algorithms internally that drive what it is that you will 
put on your bookshelf. I guess what I’m reaching for is some mechanism to bridge those and 
reconcile the two opposite ends of it. The thing is that YouTube isn’t going to expose that 
data. They’re not going to expose the hyper parameters that they make use of in order to do 
those things. Or do you think they could be forced to, in terms of algorithmic transparency, 
versus personal curation? Do you see things that can be pushed on, in order to come up with a 
way in which those two things can be understood, not as completely distinct artefacts, but as 


opposite ends of a spectrum that people can reside within at any other point? 


Jad Esber: Yeah. You touch on an interesting tension. I think there are two things. One is, 
things being built, being composable, so people can build on top of them, and can audit them. 
So, I think the YouTube algorithm, being one example of something that really needs to be 
audited, but also, if you open it, it allows other people to take parts of it and build on top of it. 
I think that’d be really cool and interesting. But it’s obviously completely orthogonal to 
YouTube’s business model and building moats. So composability is sort of one thing that 
would be really interesting. And auditing algorithms is something that’s very discussed in this 
space. But I think what you’re touching on, which is a little bit deeper, is this idea of 
algorithms not capturing emotions, and not capturing the softer stuff. And a lot of folks think 
and talk about an emotional topology for the Web. When we think about our bookshelf, there 
are memories, perhaps, that are associated with these books, and there are emotions and 
nostalgia, perhaps, that’s captured in that display of things that we are organising. And that’s 
not really very easy to capture using an algorithm. And it’s intrinsically human. Machines 
don’t have emotions, at least not yet. And so, I think that what humans present is context and 
that’s emotional context, nuance, that isn’t captured by machine curation. And so, that’s why, 
in the presentation, I talk a little bit about the pairing of the two. It’s important to scale things 
using programmatic algorithms, but also humans make it real, they add that layer of emotion 


and context. And there is this parable that basically says that human curation will end up 


267 


leading to a need for algorithmic curation. Because the more you add and organise, the more 
there’s a need for then a machine to go in and help make sense of all the things that we’re 
organising. It’s an interesting pairing, what balance is important, and it’s an open question. 
Frode Hegland: Yeah, Fabien, please. But after that, Brendan, if you could elaborate on what 
you wrote in the chat regarding this, that would be really interesting. 

Fabien Benetou: It’s to pitch something to potentially consider linking with your platform, 
it’s an identity management targeting mostly VR, at least at first. And there is completely 
federated and open source. The thing is it’s very minimalist. It just provides an identity. And 
you have, let’s say, a 3D model and a name and a list of friends. I think that’s it. But if you 
were to own things, and you were to be able to either share or display them across the 
different platforms, I think it could be quite interesting. Because, in the end, we discussed this 
quite a bit, so I’m going to go back, but there is also a social or showcasing aspect to creation 
we want to exchange. Honestly, when I do something that I’m proud of, first thing I want to 
do is to show someone. I’m going to see if my better half is around, she’s not going to get it, 
but still, I can’t stop myself, I want to show it. I have a friend, they’ ll get it, hopefully. I want 
to show you also here. And so, I want to build, and I want to show it. And I imagine a lot of 
the creation is, as soon as you find something beautiful, it’s like, “No, I don’t want to keep it 
to myself. I want to share with my people.” So, I’m wondering at which point that could also 
help this kind of identity platform or solution, because they were quite abstract in the sense 
that they’ re not specific, let’s say, to one platform, they are on top of that. But then people 
think, ‘““What for?” Okay, I can log in with, let’s say, Facebook or Apple. I know them. I trust 
them. So that’s it. I’m just going to click on that button. But it’s always a way for the identity 
maybe, like again, the discussion we had here is, my identity, me also, what I showcase 
around me that define me, and I want to not just share it to establish myself as, but also help 
others discover. So maybe it could be interesting to check how there could be a way to be 
more than an identity. 

Jad Esber: Totally if you think about DJs, their job is essentially, their profession is 
essentially to curate music and stitch things together. There are professions that centre around 
helping other people discover, and that that becomes work, right? So I think helping other 
people discover can be considered something that gives you back status or gives you back 
gratification in some form. Perhaps, it just makes you happier. But it also could give you back 
money and that it’s a profession. Arts curators, DJs. So, there’s a spectrum as well, I think a 
lot of folks will recommend it because they like it. They will recommend it because gives 
them some level of status. At the end of the spectrum, it becomes a job. Which I think is 
certainly an interesting proposition, is like, what does it look like if internet curators are 


recognised as professionals? Could there be a world where people who are curating high 


268 


value stuff could be paid? And I think, Brendan alluded to this briefly, beyond just adding 
links, like the synthesis, the commentary is really valuable, especially with the overload that 
we have today. And so, I think I alluded to this idea of invisible labor, curation being invisible 
labor. What if it was recognised? And what if it became a form of paid work? I think that 
could also be very interesting as an extension to your thought around curating to help others. 
Fabien Benetou: So, sorry. I'll just bounce back because it’s directly related, but I’m just 
going to throw it out there. If someone wants to tour through WebXR and have some of their 
favourite spaces and give me a bit of money for doing it, I’m up for attempting that. I know 
exactly how, but I think it could be quite interesting to have a tour together, and maybe put in 
our backpack whatever we like, or with whom we connect. And again, across platforms, not 
Just one. 

Jad Esber: Totally, yeah. There is precedent to that in a way, like galleries, and museums are 
institutionalised, like spaces of curated works. We pay to enter them. Is there a way where we 
can bring that down to the individual, right? A lot of the past version of the Web is taking 
institutionalised things and making them user-generated. Is there a version of galleries or 
museums that are user-generated and owned? And that’s an exploration that we’re interested 
in, as well, at “Koodos’. So, something we’re exploring. 

Frode Hegland: Fabien, I saw you put a link here to web.immers.space. Reminds me to 
mention to you guys that someone from ‘Immersed’, the company that makes the virtual 
screens in Oculus will be doing a hosted meeting soon. On a completely different tangent 
from what this is about, but I just wanted to mention to you guys. Brendan, would you mind 
going further about what you’re talking about? 

Brendan Langen: Sure. I think it’s minimal, but the act of curation, I suppose, I should have 
qualified the type of research that I’m talking about. My background is in UX research. So, 
when you’ re digging into any one of our experiences with a tool, and we run into a pinpoint, 
or we stop using, we leave the page. The data can tell us, we were here when this happened. 
But it takes so much inference to figure out what it actually was that caused it. Could be that 
we just got a phone call, and it was not a spam call for once, and we’re thinking, “Oh, wow. I 
have to pick this up and talk to my mother.” Or it could be that this is so frustrating, and as I 
kept clicking, and clicking, I just got overwhelmed, and I didn’t want to deal with it anymore. 
And everything between there. And that’s really where the role of user research comes in. 
And that was the comparison to curation, is that, we can only understand what feeling 
someone had, when they heard that song that changed their life, or read a passage that 
triggered a thought that they then wrote an essay out. And it’s something that I have to dive 
into further, and further. It’s like, the human is needed in the loop at all times. Mark and I 


have talked a lot about this. It does not matter how your data comes back to you, regardless, 
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you’re gonna need to clean it. And you’re going to need to probe into it, and enrich it with a 


human actually asking questions. 


Jad Esber: Totally, yeah. That resonates very deeply. And I can share a little bit about 
‘Koodos’, because I’ve alluded to it, but I will also share that it’s very early, and very 
experimental. So that’s why there isn’t really that much to share. But I think it centres around 
that exact idea of, how can we bottle or memorialise the feeling that we have around 
discovering that thing that resonated. And the experience, right now, centres on this idea of, 
“Hey. When I’m listening to this song, or I’m reading this article, or watching this video, and 
it resonates. What can I do with it to memorialise it, and to keep it, and to kind of create 
something based on it?” And so, right now, people create these cards that sort of link out to 
content that they love from across the Web. And on those cards, they can add context or 
commentary. And a lot of what people are adding tends to be emotional. The earliest 
experiment centred on people adding emojis, just emoji tags to the content to summarise the 
vibe of the content. And these cards are all time-stamped, so there’s also a way for you to see 
when someone came across something. And they’re all added to a library, or an archive, or a 
bedroom, or bookshelf, whatever you’re going to call it, that aggregates all the cards that 
you’ve created. So it becomes a way for you to explore what people are interested in. What 
they’re saying and feeling about the things that they come across that resonates. The last 
thing I'll share, as well, is that these cards unlock experiences. So, if I created a card for 
Brendan’s paper, for example, Ill get access to a collection, where other people have created 
cards for Brendan’s work live, and I can see all of what they commentated and created, and 
who they are, and maybe go into their libraries and see what it is that they are creating cards 
for. So, that’s the current experience. And again, in the early stages. Most of our users are 
quite young, that’s why I sort of speak a lot about identity formative years, when you’re 
constructing your identity being a really important phase in life. And so, our users are around 
that age. And that’s what we’re doing and we’re thinking about. And just provide some 


context for a lot of the perspectives that I share. 


Brendan Langen: | have to comment. I love the idea of prompting reflection. Especially at a 
stage where you are identity-forming. There’s nothing like cultivating your taste by actually 
talking about what you liked and disliked about something. And then, being able to evoke 
that in the frame of, how it made me feel in a moment, can build up a huge library of personal 
understanding. So, that’s rather neat. I need to check this out a little further. 

Jad Esber: Totally, yeah. We can chat further. I think the one big thought that has come 
about, from the early experimentation is that, people use it as a form for mental health 
reasons. Prompting you to reflect, or capture emotion over time, and archiving what has 


resonated, and what you felt over time is a really healthy thing to do. So that was an 
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interesting outcome of the early product. 


Closing Comments 


Frode Hegland: There are so many opportunities with multiple dimensions of where this 
knowledge can go. We also have, upcoming, Phil Gooch from Scholarcy, who will be doing a 
presentation. He doesn’t do anything with VR, AR or anything. But what he does do is, 
scholarcy.com analyses documents, academic documents. So they do all kinds of stuff that 
seems to be on more of the logical side, where it seems, Jad, you’re more of the emotional 
side. And I can imagine, specifically for this community, the insane amount of opportunities 
for human interactions in these environments. And then how we’re going to do the plumbing 
to make sure it is vulnerable. You said earlier, when defining Web3.0, one of the terms is 
ownable. The work we’ve been doing with Visual-Meta is very much about, we need to be 
able to own our own data. So, it was nice to hear that in that context. We’re winding down. 
It’s really nice to have two hours, so it’s not so rushed. So we can actually listen to each 
other. Are there any closing comments, questions, suggestions, or hip-hop improvisations? 
Fabien Benetou: I’m not going to do any hip-hop improvisation, not today at least. Quick 
comment, though is, I wouldn’t use such a platform. And also, I would say, without actually 
owning it, meaning for example, at least a way to export data, and have it in a meaningful 
way And I don’t pour my life into things, because especially here, is the emotional aspect 
without some safety, literal safety of being able to extract it, and ideally live, because I’m a 
programmer. So, if I can tinker with the data itself, that also makes it more exciting for me. 
But I do hope there is some way to easily, conveniently do that and hopefully, there is a need 
to consider leaving the platform. Tinkering I think it’s always worthwhile. No need to leave, 
but it’s still being able to actually have it do whatever you want. I think is pretty precious. 
Jad Esber: Yes, thank you. Thank you for sharing that, Fabien. And absolutely. That’s a very 
important consideration. So, the cards you create are tied to you, not to the space that you 
occupy or you create on ‘Koodos’. That’s a really key part of the architecture. And I hear you 
on the privacy and safety aspect. Again, this is a complex human system and so, when 
designing them, beyond the software you’re building, I think the social design is really 
important. And aspects of what is in the box, that’s for yourself. The trinkets that you keep to 
yourself, versus the cards that are the books that you present to the rest of the folks that come 
into your space. I think is an important design question. So, yeah. Thank you for sharing, 
Fabien. 

Fabien Benetou: A quick little thing, that is a lot more open, let’s say, unfortunately, I can’t 


remember the name, but three or four years ago, there was a viewer experience done by 
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Lucas something, maybe somebody will remember, where you had like a dozen or two 
dozens of clouds on top of your head, couple of scenes, and you could pull a cloud, in order 
to listen to someone else’s voice. And each space, virtual space was a prompt to, when is the 
last time you cried? Yes, www.lucasrizzotto.com. And so, his experience must be there in his 
portfolio, is three or four years old. But maybe half a dozen different spaces, with different 
ambiance, different visuals, and sounds. And every time prompting, well, I don’t know, 
what’s the meaning of life, simple, easy questions. And then, if you want to talk, you can talk 
and share it back with the community. And if you don’t want to talk. you don’t have to. So, 
it’s not what you do, but I think there are some connections, some things could be inspiring, 
also, to check it out. 

Jad Esber: I guess, on my part, I just want to say thank you for the conversation, and for 
being here for the two hours. It’s a long time to talk about this stuff. But I appreciate it. And 
yeah, I look forward to, hopefully, joining future sessions, as well. Sounds like a really 
interesting string of conversations. And it’s great to connect with you all virtually and to hear 
your questions and perspectives. Yeah, thank you. 

Frode Hegland: Yeah. It’s very nice to have you here. And the thing about the group is, 
okay, we are today, except for Dene, we’re all male and so on. But we do represent quite a 
wide variety of mentalities. And this is something we need to increase as much as we can. It 
is crucial. And also, I really appreciate you bringing in, literally, a new dimension dealing 
with emotions and identities into the discussion. So, it’s going to be very interesting moving 
forward. I was not interested in VR, AR at all in December. And then, Brandel came into my 
life. And now it is all about, I’m actually decided I can use the word metaverse because Meta 
doesn’t own it, I’ve decided to settle down. But the point is, I feel we’re already living in the 
metaverse. We’re just not seeing it through as many rich means as we can. And I don’t want 
to go into the metaverse with only social and gaming. And today, thank you for highlighting 
that we need to have our identities managed in this environment, and taken with us. So, I’m 
very grateful. And I look forward to seeing those of you who can on Friday. And we’re going 
to be doing, as I said, every two weeks presentations in this format. And yes. Anything else 
before I rush off and make some dinner for the family? 

Fabien Benetou: I have a quote for this. It’s on my desktop, actually. It’s, “When technology 
shifts reality, will we know the world has changed?” it’s from Ken Perlin that we mentioned 


last time. I'll put it in the chat. 
Frode Hegland: Very nice indeed. Thanks for that. 
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https://youtu.be/2NeSCOrVw24?t=1353 


Gavin Menichini: Immersed is a virtual reality product, working productivity software, 
where we make virtual offices. And so, what that means is, Immersed is broken down into 
two categories, in my opinion. We have a solo use case, and we have a collaboration meeting 
use case. So, the main feature that we have in Immersed is the ability to bring your computer 
screen, whether you have a Mac, a PC, or Linux, into virtual reality. So, whatever is on your 
computer screen is now brought to Immersed. And we’ve created our own proprietary 
technology to virtualize extensions of your screen. Very similar to, if you had a laptop or 
computer at your desk, and you plugged in extra, physical monitors, from our screen real 
estate. We’ve now virtualized that technology. It’s proprietary to us. And we’re the only ones 
in the world who can do that. And then, now at Immersed, instead of you working on one 
screen, for example, I use the MacBook Pro for work, so instead of me working on one 
MacBook Pro, with an Oculus Quest 2 headset, or a compatible headset, I can connect it to 
my computer, have a Immersed software on my computer, in my headset, bring my screen 
into virtual reality, have the ability to maximize it to the size of an iMac screen. I can shrink it 
and then create up to five virtual monitors around me for a much more immersive work 
experience for your 2D screens. And you can also have your own customized avatar that 
looks like you, and you can beam into all these cool environments that we’ve created. Think 
of them as higher fidelity, higher quality video game atmospheres. But not like a game, more 
like a professional environment. But we also have some fun gaming environments, or space 
station offices, or a space orbitarium, auditorium. We have something called alpine chalet, 
like a really beautiful ski lodge. Really, the creativity is endless. And so, within all of our 
environments, you can work there, and you can also meet and collaborate with people as 
other avatars, instead of us meeting here on zoom, where we’re having a 2D, very 
disconnected experience. I’m sure each of you probably heard the term Zoom fatigue or 
video conference fatigue? That’s been very real, especially with the COVID pandemic. And 
so, fortunately, that’s hopefully going away, and we can have a little bit more in-office 
interactions. But we believe Immersed is the perfect solution for hybrid and remote working. 
It’s the best tech bridge for recreating that sense of connection with people. And that sense of 


connection has been very valuable for a lot of organizations that we’re working with, as well 
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as enhancing the collaboration experience from our monitor tech, and our screen sharing, 
screen streaming technology. So, people use it for the value, and the value that people get out 
of it is that, people find themselves more productive when working in Immersed, because 
now, they want to have more screen real estate, like all the environment we’ve been 
potentially created, to help preach cognitive focus. So, I have lots of news for customers and 
users who tell us that when they’re Immersed. They feel hyper-focused. More productive. In 
a state of deep workflow, whatever term you want to use. And people are progressing through 
the work faster, and feel less distracted. And then, just also, generally more connected, 
because when you’re in VR, it really feels like you have a sense of presence when you’re 
sitting across from a table from another avatar that is your friend or colleague. And that really 
boosts employee and person satisfaction, connection, just for an overall engaging, better 
collaborative experience when working remotely. Any questions around what I explained, or 


what Immersed is? 


Dialogue 
https://youtu.be/2NceSCOrVw24?t=1549 


Fabien Benetou: Super lovely. When you say screen sharing, for example, here I’m using 
Linux. Is it compatible with Linux? Or is it just Windows or macOS? Is it web-based? 
Gavin Menichini: So, it is compatible with Linux. And so, right now, you can have virtual 
monitors through a special extension that we’ve created. We’re still working on developing 
the virtual display tech to the degree we have for Mac and Windows. Statistics says that 
Linux is only one of two percent of our user base. And so, for us, as a business, we obviously 
have to optimize for most of our users. Since we’re a venture-backed startup. But that’s 
coming in the future. And then, you can also share screens with Linux. And so, with some of 
the extensions, you can use it for having multiple Linux displays, you can share those 
screens, as well, within Immersed. 

Video: https://youtu.be/2NcS5COrVw24?t=1594 


Alan Laidlaw: That’s great. Yeah, this is really impressive. This is a question that may be 
more of a theme to get into later. But I definitely see the philosophy of starting with, where 
work is happening now, and like the way that you make train tracks, bringing bits and pieces 
into VR so that you can get bodies in there. I’m curious as to, once that’s happened or once 
you feel like you’ve got that sufficiently covered, is there a next step? What would you want 
the collaborative space in VR to look like that is unlike anything that we have in the real 
world, versus... Yeah, I’d love to know where you stand philosophically on that, as well, as 


whatever the roadmap is? 
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Gavin Menichini: Sure. If I’m understanding your question properly, it’s how do we feel 
about how we see the evolution of VR collaboration, versus in-person collaboration? If we 
see there’s going to be an inherent benefit to VR collaboration as we progress, versus in 
person? 

Alan Laidlaw: Yeah, there’s that part. And there’s also, the kind of, is the main focus of the 
company to replicate and provide the affordances that we currently have, but in VR? Or is the 
main focus, now that you know once we’ve ported things into a VR space, let’s explore what 
VR can do? 

Gavin Menichini: Okay. So, it’s a little bit of both. It’s mostly just, we want to take what’s 
possible for in-person collaboration and bring it into VR, because we see a future of hybrid 
remote working. And so, COVID, obviously, accelerated this dynamic. So, Renji, our 
founder, started the company in 2017, knowing, believing that hybrid remote work was gonna 
become more and more possible as the internet and all things Web 2.0 became more 
prevalent. And we have technology tools where you don’t have to drive into an office every 
single day to accomplish work and be productive. But we found that the major challenges 
were, people aren’t as connected. The collaboration experience isn’t the same as being in 
person. So those are huge challenges for companies, in a sense of a decrease in productivity. 
So, all these are major challenges to solve. And those are the challenges that Renji set out to 
go build and fix with Immersed. So when we think about the future, we see Immersed as the 
best tech bridge, or tool for hybrid or remote working. Where you can maximize that sense of 
connection that you have in person, by having customizable avatars, where fidelity and 
quality will increase over time, giving you the tech tools through multiple monitors and solo 
work. Enhancing the solo work experience. So people become more productive, which is the 
end goal of giving them more time back in the day. And then also, corporations can continue 
to progress, as well, in their business goals, while balancing that with giving employees more 
time back of their day to find that beautiful balance. And so, we see it as a tech bridge, but 
we, as a VR company, we’re also are exploring the potentials of VR. Is there something that 
we haven’t tapped into yet that could be extremely valuable for all of our customers and users 
to add more value to their life and make their life better? So, it’s less so that, it’s more so we 
want to virtualize, make the hybrid remote collaboration, work experience, much more full, 
better value, with more value than it currently exists today with the Zoom, Slack, Microsoft 
Teams paradigm. 

Brandel Zachernuk: Yeah, I’m curious. It sounds like, primarily, or entirely, what you’ve 
built is the the connective tissue between the traditional 2D APPs that people are using within 
their computer space, and being able to create multi-panels, that people are interacting with 


that content on. Is that primarily through traditional input? Mouse, keyboard, trackpad? Or is 
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this something where they’re interacting with those 2D APPs through some of the more 
spatial modalities that are offered hands or controllers? Do you use hands or is it all entirely 
controller-based? 


Gavin Menichini: Yeah, great question. So, the answer is, our largest user base is on the 
Oculus Quest 2. It’s definitely the strongest headset, bang for your buck on the market for 
now. There’s no question. But, right now, you can control your VR dynamics with the 
controllers or with hand tracking. We actually suggest people use hand tracking, because it’s 
easier, once you get used to it. One of the challenges we face right now is, there is an inherent 
learning curve for people learning how to interact with VR paradigms. And, as me being on a 
revenue side, I have to demonstrate Immersed to a lot of different companies and 
organizations, and so it can be challenging. At some point, I imagine it would be very similar. 
And I was born in 95, and so I wasn’t around these times. But I imagine it feels like demoing 
email to someone for the first time, on a computer, and they’ve never seen a computer, where 
they totally understand the concept of email. No more paper memos, no more post-it notes. 
Paper organization and file cabinets, all exist in the computer, and they get it. But, when I put 
a computer in front of them for the first time, they don’t know how to use it. What’s this 
track? They had the keyboard, the mouse, they don’t understand the UI, UX of the Oculus, 
the OS system. They don’t understand how to use that, so it’s intimidating. So, that’s the 
challenge we come across. And then, that answers your point with your first question, 
Brandel? 


Brandel Zachernuk: Yeah, I’ve got some follow-ups, but I’Il cede the floor to Frode. 


Frode Hegland: Okay. I’m kind of on that point. So, I have been using Immersed for a bit. 
And the negatives, to take that first, is that I think the onboarding really needs help. It’s nice 
when you get that person standing to your side and pointing out things, but then... So, the 
way it works is, the hand tracking is really good. That is what I use. I use my normal 
keyboard, physical keyboard on my Mac, and then I have the monitor. But it’s, to me, a little 
too easy to go in and out of the mode where my hands change the position and size of the 
monitor. You’re supposed to do a special hand thing to lock your hands to not be doing that. 
And so there’s pinning. So, when you’re talking about these onboarding issues, that’s still a 
lot of work. And that’s not a complaint about your company. That’s a complaint across the 
board. The surprise is also, it really is very pleasant. I mean, here, in this group, we talk about 
you know many kinds of interactions, but what I would like, in addition to making it more 
locked, to make the pinning easier. I do find that, sometimes, it doesn’t want to go exactly 
where I want. I’m a very visual person, kind of anal in that way, to use that language. I want 
it straight ahead of me, but very often it’s a little off. So, if I resize it this way, then it kind of 


follows. So, in other words, I’m so glad that you are working on these actual realities, boots 
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on the ground thing, rather than just hypotheticals. Because it shows how difficult it is. You 
get this little control thing on your wrist, if there was one that says “hyper control mode”, 
different levels. Anyway, just observation, and question, and point. 

Gavin Menichini: Yeah. I can assure you that we obsess over these things internally. Our 
developers are extremely passionate about what we’re building. We have a very strong XR 
team. And our founder is very proud about how hard it is to get to our company, and how 
many people we reject. So, we really are hiring the best talent in the world, and I’ve seen this 
first-hand, getting to work with them. And we also have a very strong UI, UX team. But 
we’re really on the frontier of, this has never been done before. And we are pioneering. What 
does it mean to have excellent UI, UX paradigms and user onboarding paradigms in virtual 
reality? And one of the challenges we face is that, it’s still early. And so people are still trying 
to figure out, even foundations for what is good UI, UX. And we’re now introducing space, 
like spatial computing. And we’re going from 2D interfaces to 3D. What have we learned 
from good UI, UX or 2D translate to 3D, and paradigms of this? And people are now not just 
using a controller and mouse, they’re using hand tracking and spatial awareness. And how do 
we build good, not only do we understand what’s a good practice for having good paradigms 
in UI, UX, how do we code that well? And how do we build a good product around that, 
while also having dependencies on Oculus, HTC, and Apple? Where we’re dependent upon 
hardware technology to support our software. So we still live very much in the early days, 
where there’s a lot of tension of things are still being figured out. Which is why we’re a 
frontier tech. Which is why it takes time to build. But even with VR, AR, I think, it’s just 
going to take longer because there are so many more factors to consider. The people who 
pioneered 2D technology, Apple, Microsoft, etc, they didn’t have to consider. And so, I think 
the problem we’re solving candidly is exponentially harder than the problem they had to 
solve. But we also get to stand on their shoulders, and take some precedence that they built 
for us, and apply that to VR, where it makes sense. 

Brandel Zachernuk: So, in terms of those new modalities. In terms of the interaction 
paradigms that seem to make the most sense, it sounds like you’re not building software that 
people use, as much as you’re using making software that people reach through to their other 
software with, at this point. Is that correct? You’re not making a word processor, you’re 
making the app that lets people see that word process. Which is a big problem. I’m not 
minimizing it. My question is: 

Do you have observations based on what people are using the way that they’re changing, for 
example, the size of their windows, the kinds of ways that they’re interacting with it? Do you 
have either observations about what customers are doing as a result of making the transition 


into effective productivity there? Or do you have any specific recommendations about things 
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that they should avoid or reconsider given the differences in, for example, pixel density, or 
the angular fidelity of hand tracking within 3D, in comparison to the fidelity of being able to 
move around a physical mouse and keyboard? Given that those things are so much more 
precise. But also, much more limited in terms of the real estate that they have the ability to 
cover. Do you have any observations about what people do? Or even better, any 
recommendations that you make to clients about what they should be doing as a result of 
moving into the new medium? 

Gavin Menichini: Yeah, really good question. There are a few things. There’s a lot of things 
we could suggest. So, a lot of what we’re building is still very exploratory, of what’s the best 
paradigm for these things? And so, we’ve learned a lot of things, but we also understand 
there’s a lot more for us to build internally and explore. First and foremost, we definitely do 
not take, hopefully, this is obvious, but to address it, we definitely do not take a dystopian 
view of VR, AR. We don’t want people living in the headset. We don’t want people strapped 
it to their face extremities, like a feeding tube and water, etc. That’s not the future we want. 
We actually see VR, AR as a productivity enhancer, so people can spend less time working, 
because they’re getting more done in our products, because we’ve created a product so good 
that allows them to be more productive, so they get more done at work, but also, have more 
time to themselves. So, we suggest people take breaks, we don’t want you in a headset for 
eight hours straight. The same way no person would suggest for you to sit in front of your 
computer, and not stand, use the restroom, eat lunch, go on a walk or take a break. We could 
take the same paradigms. Because you can get so focused on Immersed, we also encourage 
our users to like, “Yeah, get stuff done, but take a break”. But then we’re also thinking 
through some of the observations we found. We’ve been surprised at how focused people 
have been. And the onboarding challenge is a big challenge, as Frode was mentioning. It’s 
one that we think about often. How do we make the onboarding experience better? And 
we’ve made progressions based on where we came from in the past. So, Frode, you’re seeing 
some of the first iterations of our onboarding experience, in the past, we didn’t have one. 
There’s something we actually pushed really hard for. We saw a lot of challenges of users 
sticking around because we didn’t have one. And we’re now continuing to push how do we 
make this easier. Explain things to people without making it too long, where people get 
uninterested and leave. It’s a really hard problem to solve. But we found, as we’re having an 
easier onboarding experience, helping people get used to the paradigms of working in VR 
and AR, and explaining how our technology works, and letting them get to, what we like to 
call this magic moment, of where they can see the potential of seeing and having their screens 
in VR. Having it be fully manipulative, you’re like the Jedi in the force. You can push and 


pull your screens with hand tracking, to pinch and expand. Put them all around you. If I’m 
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answering your question, Brandel, we’re still exploring a lot of paradigms. But we found that 
it’s surprising how focused people are getting, which is awesome and encouraging. We find, 
which isn’t surprising as much anymore, companies, organizations, and teams are always 
very wild at how connected they feel to each other. So we always try to encourage people to 
work together. So, even on our elite tier, which is just our middle tier, like a pro think of it as 
a pro solo user, you have the ability to collaborate with up to four people in a private room. 
But we also have public spaces, where people can hang out and it’s free to use. Just think of it 
as a virtual coffee shop. You can hang out there, and meet with people. You can’t share your 
screens, obviously, for security reasons. But you can meet new people and collaborate. And 
it’s been cool to see how we’ve informed our own community where people can be connected 
with each other to be able to hang out and meet new people. So, hopefully, that answers a 
little bit of your question. There’s still a lot more we’re learning about the paradigms of 


working in 2D screens, and what people prefer, what’s the best practice. 


Brandel Zachernuk: Yeah. One of the issues that I face when I think about where people 
can expect to be in VR productivity at this point, is the fact that Quest 1, Quest 2 and Vive, 
all of these things have a focal distance. Which is pretty distant, normally a minimum 
accommodation distance is about 1.4 meters, which means that anything that’s at 
approximately arm’s length. Which is where we have done the entirety of our productivity in 
the past. Is actually getting to within eye strain territory. The only headset that is out on the 
market that has any capacity for addressing that kind of range is actually the Magic Leap. 
Which I don’t recommend anybody pursue, because it’s got a second focal plane at 35 
centimetres. Do you know where people put those panels on Quest? On Vive? I don’t know if 
you’ve got folks in a crystal or a coral value, whether that has any distinction in terms of 
where they put them? Or alternatively, do you recommend or are you aware of anybody 
making any modifications for being able to deal with a closer focal distance? I’m really 
interested in whether people can actually work the way they want to, as a consequence of the 


current limitations of the hardware at the moment. 


Gavin Menichini: Yeah. There are a few things in response to that. One: We’ve actually 
found, internally, even with the Quest 2, although the screen distance, et cetera, focal point, is 
a challenge, we’ve actually found that people in our experience are reporting less eye strain 
working in VR, than they are working from their computer. We’re candidly still trying to 
figure out why that’s the case. I’m not sure how the distance and the optics games that they’re 
playing in the Quest 2 and other headsets we use. But we’ve actually found that people are 
reporting less eye strain, just solely on customer reviews and feedback. So we haven’t done 
any studies. I personally don’t know a lot around IPDs and focal length distance of the exact 


hardware technology of all the headsets on the market. All I’m doing is paying attention to 
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our customers, what they’re saying, and our users. And we’re actually, surprisingly, not 
getting that much eyestrain. We’ve actually said that a lot of people say they prefer working 
in VR than from their computers, without even blue light glasses. And they’re still getting 
less eye strain. So, the science and technicalities of how it’s working, I’m not sure. It’s 
definitely out of my realm of expertise. But I can assure you that the hardware manufacturers, 
because of our close relationship with Meta, HTC, they’re constantly thinking about that 
problem too, because you’re strapping an HMD to your face, how do you have a good 
experience from a health standpoint for your eyes? 

Brandel Zachernuk: Do you know how much time people are clocking in it? 

Gavin Menichini: On average, our first user session is right around an hour 45 minutes to 
two hours. And we have power users who are spending six to eight hours a day inside of 
Immersed, clocking that much time in and generating getting value out of it. And it’s 
consistent. And I’m not sure what our average session time is. I would say it’s probably 
around an hour, two hours. But we have people who use it for focus first, where they want to 
go and focus sessions on Immersed, or people will spend four or five hours in it, and our 
power users will spend six, seven, eight hours. 

Frode Hegland: | can address these few points. Because, first of all, it’s kind of nice. I don’t 
go on Immersed every week, but when I do, I do get an email that says how many minutes I 
spent in Immersed, which is quite a useful statistic. So, I’m sure, obviously, you guys have 
more on that. When it comes to the eye strain, I tend to make the monitor quite large and put 
it away to do exactly the examination you’re talking about, Brandel. And I used to not like 
physical monitors being at that distance. It was a bit odd. But since I am keyboard, trackpad, 
where I don’t have to search for a mouse, I don’t need to see my hands anyway, even though I 
can. I do think that works. But maybe, Gavin, would you want to, you said you had a video to 
share a little bit of what it looks like? 

Gavin Menichini: Sure, yeah. I can pull that up real quick. So it’s a quick marketing demo 
video, but it does do a good job of showcasing the potential of what’s possible. And I’m not 
sure if you guys will be able to hear the audio. It’s just fun background music. It’s not that 
important. The visuals are what’s more important. Let me go ahead and pull this up for us 
real quick. 

Frode Hegland: | think you can just mute the audio and then talk if you want to highlight 
something, I guess. 

Gavin Menichini: Okay. Actually, yeah. That’s probably a good idea. So, this is also on 
YouTube. So just for each of your points, if you guys are curious and want to see more 
content, just type in Immersed VR on YouTube. Our Immersed logo is pretty clear. Our 


content team and marketing team put out a lot of content, so if you’re curious. We also have a 


280 


video called “Work in VR, 11 tips for productivity”, where a head of content goes through 
some different pro tips. If you’re curious and just want to dive in more of a more nuanced 
demo of how you do things, etc, to see more of the user experience. So, this is a good, helpful 
high level video. So you can see you can have full control of your monitor. You can make it 
ginormous, like a movie screen. We have video editors, day traders, finance teams, and 
mostly developers are our main customer base. As you can see here, the user just sitting down 
at the coffee table, the keyboard is tracked. We also have a brand new keyboard feature 
coming out, it’s called keyboard passthrough, where we’ll leverage the cameras of your 
Oculus Quest to hold the VR and see your real-life keyboard, which we’re very excited 
about. And here you can just see just a brief collaboration session of two users collaborating 
with each other side by side. You can also incorporate your phone into VR, if you want to 
have your phone there. And then, here you’ll see what it looks like to have a meeting in one 
of our conference rooms. So, you can have multiple people in the room, we usually had 30 
plus people in an environment, so it can easily support that. It also depends on, obviously, 
everyone’s network strength and quality, very similar to Zoom, or phone call. And that shows 
how quality the meeting is from their audio and screen sharing input, but if everyone’s on a 
good network quality, that’s not an issue. And then, lastly here, you can see one of our users 
with five screens, working in a space station. And that’s about it. Any questions or things that 


stood out from that, specifically? 


Frode Hegland: Yeah. A question about the backgrounds. You have some nice environments 
that can be applied. I think we can also import any 360 images, is that right, currently? And if 
so, can we also load custom 3D environments in the future? Are you thinking about 


customization for that aspect of it? 


Gavin Menichini: Yes. So, we are thinking about it, and we do have plans for users to 
incorporate 3D environments. There are a few challenges with that, for a few obvious 
reasons, which I could touch on a second. But we do support 360 environments, 360 photos 
for users to incorporate. And we also have a very talented artist and developer team that are 
constantly making new environments. And we have user polls, and we figure out what our 
users want to build and what they'd like to see. And as we, obviously, continue to grow our 
company, right now we’re in the process of fundraising for a series, and once we do that, 
we’re hoping to go from 27-28 employees right now, to at least 100 by the end of the year. 
The vast majority of them will be developers to continue to enhance the quality of our 
product. And then, we also will support 3D imports of environments. But because the Quest 2 
has some compute limitations, we have to make sure that each of our environments have 
specific poly counts, and specific compute measurements, so that the Quest 2 won’t explode 


if they try and open that environment in Immersed, as well as making sure that your 
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Immersed experiences can be optimized in high quality and not going to lag, et cetera. So 
right now, we’re thinking: How do we enable our users to build custom environments? And 
then, two: How do we make sure they meet our specific requirements for the Quest 2. But 
naturally, over time, headsets are getting stronger, computing powers are getting better. Very 
similarly when you go from a Nintendo 64 graphics, to now the Xbox Series X. The 
ginormous quality. Headset quality will be the same. So, we’ll have more robust 
environments to have some more, give and take optimizations for environments our users 
give to us. So it isn’t our pipeline, but we’re pushing it further down the pipeline than we 
originally wanted. Just doe to some natural tech limitations. And also the fact that we are an 
adventure back startup, and we have to be extremely careful of what we work on, and 
optimize for the highest impact. But we’re starting to have some more fun and having some 
traction in our series A conversations. And hopefully have some more flexibility, financially, 


to continue pushing. 


Alan Laidlaw: Yes. So, this is maybe a, kind of, Twilio-esque question about the design 
material of network strength bandwidth and compute, like you mentioned. And I’m 
wondering, I saw in the demo, the virtual keyboard that, of course, the inputs would be 
connected to a network versus a physical keyboard that you already have in front of you, if it 
were possible to use the physical keyboard and have those inputs go into the VR 
environment, or AR environment, in this case, would that be preferred? Is that the plan? And 
if so, you know, that opens up, I mean, this is such a rich pioneer, as you mentioned, territory, 
so many ways to handle this. Would there be a future where, if my hands are doing one thing, 
then that’s an indication that I’m in my real world environment, but if I hand at something 
else and that’s suggesting, you know, take my hand into VR, so I can manipulate something? 
I’m curious about. Any thoughts about, essentially, that design problem, versus the hard 
physical constraints of bandwidth? Is it just easier? Does it make a better experience to stick 
with a virtual keyboard for that reason? So, you don’t, at least, have a disconnect between 


real world and VR? And I’m sure there are other ways to frame that question. 


Gavin Menichini: No, that’s fine. And I can answer a few points and a few follow up 
questions to make sure I understand you correctly. For the keyboard, specifically, the current 
keyboard tracking system we have in place is not optimal. It was just the first step of what we 
wanted to build to help make the typing VR problem easier, which is our biggest request. So 
we are now leveraging, I think, a way stronger feature, which is called “Keyboard pass- 
through”. So, for those who you know, the Oculus Quest 2 has a pass-through feature, where 
you can see the real world around you through the camera system, and they’re stitching the 
imagery together. We now have the ability to create a pass-through portal system, where you 


can cut out a hole in VR over your keyboard. So, whatever keyboard you have, whether it’s 
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Mac, Apple, whatever. The funky keyboards, that a lot of our developers really like to use for 
a few reasons, you can now see that keyboard in your real hands through a little cut-out in 
VR. And then, when it comes from inputs, of what you mentioned of doing something with 
your hands, it being a real life thing versus VR thing. Are you referring to that in regards to 
having a mixed reality headset where it can do AR and VR and you want to be able to switch 


from real world to VR with the hand motion? 


Alan Laidlaw: Yeah. A piece of my question. I can clarify. I am referring to mixed. But 
specifically where that applies is the cut-out window approach, is definitely a step in the right 
direction. But it seems that’s still based entirely on the Oculus understanding of what your 
fingertips are doing. Which will obviously have some misfires. And that would be an 
incredibly frustrating experience for someone who’s used to a keyboard always responding, 
hitting the keys that you’re supposed to be hitting. So, at some point, it might make more 
sense to say, “Okay, actually we’re going to cut out. We’re going to forget the window 
approach and have the real input from the real keyboard go into our system”. 

Gavin Menichini: So, that’s what it is, Alan. Just to further clarify, we always want our users 
to use their real hands on the real keyboard. And you’re not using your virtual hands on a 
virtual keyboard. You’re now seeing, with pass-through, your real hands and your real 
keyboard, and you’re typing on your real keyboard. 

Frode Hegland: A really important point to make in this discussion is, if for a single user, 
there are two elements here: There is the thing around you image of 3D, and then you have 
your screen. But that is the normal Mac, Linux or Windows screen. And you use your normal 
keyboard. So, I have, actually, used my own software. I’ve used Author to do some writing 
on a big nice screen, so it is exactly the keyboard I’m used to. 

Alan Laidlaw: Right. So, how that applies to the mixed reality question is, if I’m using the 
real keyboard, have the real screen, but one of my screens is an iPad, a touch screen, that’s in 
VR, where I want to move some elements around, how do I then, transition from my hands in 
the real world to now I want my hand to be in VR? 

Gavin Menichini: So, you’re going to be in Immersed, as of now. You’re going to be in VR, 
and you’re going to have a small cut out into the real world. And so, it’s just, right here is a 
real world, through a cutout hole, and then, if you have your hands here, and you want to 
move your hands into here, the moment your hands leave the pass-through portal in VR, it 
turns into virtual hands. And so, to further clarify, right now, your virtual hands, you have in 
hand tracking, will still be over your hands on the pass-through window. We’re 
experimenting taking that out for further clarity of seeing your camera hands on your 
keyboard. But, yes. When you’re in Immersed, it’1l transition from your camera hands, real 


life hands, to virtual hands. If you have an iPad and you want to swipe something, whatever, 
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it’s that’s seamless. But then, for mixed reality dynamics, in the future, we’re not sure what 
that’s going to look like, because it’s not here yet. So, we need to experiment, figure out what 
that looks like. 


Fabien Benetou: Yeah, thank you. It’s actually a continuation of your question because you 
asked about the background environment using 360, and including the old model. It’s also a 
question that you know I was going to ask, and I guess Gavin did, because I’m a developer, 
you can imagine it too. If it’s not enough, if somehow there are features that I want to 
develop, and they are very weird, nobody else will care about it, and, as you say, as a start-up 
you can’t do everything, you need to put some priorities. What can I do? Basically, is it open 
source? if not, is there an API? If there is an API, what has the community built so far? 
Gavin Menichini: Yeah, great question. So, as of now, we currently don’t have any APIs or 
open SDKs, open source code for users to use. We’ve had this feature request a lot. And our 
CEO is pondering what his approach wants to be in the future. So, we do want to do 
something around that in the future. But, because we’re still so early stage, and we have so 
many things we have to focus on, it’s extremely important that we’re very careful with what 
we work on, and how focused, and how hard working we are towards those. As we continue 
to progress as a company, and as our revenue increases, as we raise subsequent rounds of 
funding, that gives us the flexibility to explore these things. And one of the biggest feature 
requests we’ve had is having an Immersed SDK for our streaming monitor technology so 
people can start to play with different variations of what we’re building. But I do know that 
Renji does not allow for any free, open source coding work whatsoever. Just for a few 
reasons legality-wise, and I think we had a few experiences in the past where we experiment 
with that, and it backfired to where developers were claiming they owed, they deserved 
equity, or funding. It was a hot mess. So, we don’t allow anyone to work for us for free, or to 
give us any form of software, to any regard, any work period, to prevent any legal issues, to 
prevent any claims like that ,which is kind of unfortunate. But he’s a stickler and definitely 
will not budge on that. But in the future, hopefully, we’ ll have an SDK or some APIs that are 
opened up, or open source code, once we’re more successfully established for people to 
experiment and start making their own fun iterations to immerse on. 

Brandel Zachernuk: | have a question about the windows. You mentioned that, when 
somebody has a pro subscription, they can be socially connected, but not share screens. I 
presume, in an enterprise circumstance, people can see each other’s windows. Have you 
observed any ways in which people have used their windows more discursively, in terms of 
having them as props, essentially, for communicating with each other, rather than primarily, 
or solely for working on their own? The fact that they can move these monitors, these 


windows around, does that change anything about the function of them within a workflow or 
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a discussion context? 


Gavin Menichini: Yeah. So, to clarify under the tier and your functionality. We have a free 
tier, where you can connect your computer and traverse the gap. You get one free virtual 
display. You cannot, on a free tier, ever share screens in all of our public rooms. You can’t 
share screens, regardless of your license. Here, the only place you can share screens is ina 
private collaboration room. Which means, you have to be on our elite tier, or a teams tier. On 
our elite tier, which is our mid-pro-solo tier, you can have up to three other people in the 
room with you, four total, and you can share screens with each other. And the default is, your 
screens are never shared. So, if you have four people in a room, and they each have three 
screens up, you cannot see anyone else’s screen until you voluntarily share your screen and 
confirm that screen. And then, it will highlight red, for security purposes. But if you’re an 
environment where, Brandel, you wanted to share your screen, when you share your screen 
and say, we’re all sitting at a conference room table, if I have my screens like, one, two, 
three, right here, and I share my middle screen, my screen is then going to pop up in your 
perspective to you. To where you have control of my shared screen. You can make it larger. 
Make it bigger. Shrink it, etc. And we’re also going to be building different environment 
anchors to where say, for example, in your conference room, and in a normal conference 
room you have a large tv on the wall, say, in virtual reality, you could take your screen and 
snap it to that place, and once it’s snapped into that little TV slot, that screen will be 
automatically shared and everyone sees it at that perspective, rather than their own 
perspective. And then, from a communication standpoint, we have teams who will meet 
together in different dedicated rooms, and then they’II share screens, and look at data 
together. There’s... I can’t remember quite the name, it’s a software development team where 
something goes down, they have to very well come together. Devops teams come together, 
they share screens looking at data to fix a down server or something, and they can all see, and 
analyse that data together. And we’re exploring the different feature adds we can add to make 


that experience easier and more robust. 


Brandel Zachernuk: And so, yeah. My question is: Are you aware of the ways in which 
people make use of that in terms of being able to share and show more things? One of the 
things about desktop computing, even in the context where people are co-located, co-present 
in physical meet space, you don’t actually have very good performability of computer 
monitors. It kind of sucks in Zoom. It kind of sucks in real life, as well. Do people show and 
share differently, as a consequence of being in Immersed? Can you characterize anything 
about that? 


Gavin Menichini: Yes. So, the answer is yes. They have the ability to share more screens, 


and so, in meet space, in real-world, a funny term there for meet space, but. You can only 
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have one computer screen if you’re working on a laptop, and that’s frustrating. Unless you 
have a TV, you have to airdrop, XYZ, whatever. But, in Immersed, you have up to five 
screens. And so, we have teams of four, and they’ I share two or three screens at once, and 
they can have a whole arrangement of data, 10 screens are being shared, and they can 
rearrange those individually so it all pops up in front of them, and then, they all rearrange 
them in order that they want, and they can all watch a huge sharing screen of data. That is not 
possible in real life because of the technology we provide to them. And then, there’s different 
iterations of that experience where, maybe, it’s two or three screens, it’s here, it’s there. And 
so, because of the core tech that we have where you can have multiple screens and then share 
each of those, that opens up the possibility for more data visualization, because you have 
more screen real estate. This opportunity to collaborate more effectively, and if you had one 
computer screen on Zoom, which as you mentioned, is challenging, or even in real life, 
because in real life you could have a computer and two TVs, but in Immersed you could have 


eight screens being shared at once. 


Brandel Zachernuk: And do you share control? Is it something where it’s only the person 
sharing it has the control, so other people would have read-only access? Or do you have the 
ability for people to be able to pass that control around? Send the user events such that 
everybody would be able to have shared control? 

Gavin Menichini: So, not right now, but we’re building that out. For the time being, we want 
everyone just to use collaboration tools they are currently using. Use Google Docs. Use Miro. 
Use Slack. Whatever. So, the current collaboration documents you guys are using now, we 
just want to use those applications on Immersed, because whatever you can run on your 
computer, you can run on your screen in Immersed. It is just your computer in Immersed. So, 
we tell people to do that. But now they get the added benefit of deeper connection. Just 
actually to be sitting next to your employee, or your colleague and then, now you can have 
multiple screens being shared. So, now it’s like a supercharged productivity experience, 
collaboration experience. Any other questions? I have about four minutes left, so I want to 


make sure I can answer all the questions you guys have. 


Fabien Benetou: I’|l make a one minute question. I’ll just say faster. If I understood 
correctly, the primitive is the screen. But is there anything else beyond the screen? Can you 
share 3D assets? Would the content can be pulled from the screen? If not, can you take 
capture of the screen. either as image, or video? And is it the whole screen only or part of the 
screen? And imagining you’ve done that, let’s say, part of the screen as a video of 30 seconds, 
can you make it permanent in the environment so that if I come back with colleagues 
tomorrow? Capture? Because that’s the challenge we have here all the time, we have great 


discussions and then, what happens to the content? 
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Gavin Menichini: So, it’s in our pipeline to incorporate other assets that will be able to be 
brought into Immersed, and then remain persistent in the rooms. So, we’ve created the 
technology for persistent rooms, meaning, whatever you leave in there, it’s going to stay. 
Very similar to a conference room that you’ ve dedicated for project. You put post notes 
around the wall, and obviously, come back to it the next day. So there same concept when in 
VR. And then, we also have plans to incorporate 3D assets, 3D CAD models, et cetera, into 
Immersed. But because you have your screens and teams are figuring out how to collaborate 
on 2D screens, we’re just, for the time being, we’re saying just continue to use your CAD 
model software on your computer 2D. But in the future we’ll have that capability. We also 
don’t want to be like F3D modelling VR software. So, we’re trying to find that balance. 
Which is why it’s been de-prioritized. But it is coming. And hopefully, in 2022 and then, we 
have also explored having video files that are in form of screens, or an image file, or post-it 
notes, We’re also going to improve our whiteboard experience, which is just some of one of 
our first iterations. And so, there’s a lot of improvements we’re going to be making in the 
future, in addition to different assets, photos, videos, 3D modelling software, et cetera. We’ve 
had that request multiple times and plan on building it in the future. 

Fabien Benetou: Oh, and super quick. It means you get in, you do the work, you get out, but 
you don’t have something like a trace of it as is right now? 

Gavin Menichini: As in persistence? As in you get in, you leave your screens there? 

Fabien Benetou: Or even something you can extract out of it. Frode was saying that, for 
example, he gets an email about the time he spent on a session, but is there something else? 
Again, because usually, you have maybe another eureka moment, but you have some kind of 
realization in the space, thanks to the space and the tools. And how can you get that it’s really 
a struggle. 

Gavin Menichini: I’m not sure, I’m sorry. I’m not sure I’m understanding your question 
correctly, but well, so it’s... 

Brandel Zachernuk: Maybe I can take a run of it. So, when people play VR games, at a VR 
arcade, one of the things that people will often produce is a sizzle reel of moments in that 
action. There’s a replay recording, an artifact of the experience. Of that process. 

Gavin Menichini: Okay, yes. So, for the time being there is no functionality in Immersed for 
that. But Oculus gives you the ability to record what you’re watching in VR. And you can 
pull that out and take that experience with you, as well as take snapshots. And then, we have 
no plans on incorporating that functionality into Immersed because Oculus has it, and I think 
HTC does, and other hardware manufacturers will provide that recording experience for you 
to then take away with you. 


Frode Hegland: Thank you very much, Gavin, a very interesting, real-world perspective on a 
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very specific issue. So, very grateful. We'll stay in touch. Run to your next meeting. When 
this journal issue is out, I’1l send you an update. 
Gavin Menichini: Thank you, Frode. It was a pleasure getting to chat with each of you. God 


bless. Hope you guys have a great Friday, weekend, and we’ll stay connected. 
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Further Discussion 


https://youtu.be/2NceSCOrVw24?t=3987 


Frode Hegland: Oh, okay. That sounds interesting. Yeah, we can look at changing times and 
stuff. So, briefly on this, and then on the meeting that I had with someone earlier today. This 
is interesting to us, because they are thinking a lot less VR than we are. But it is a real and 
commercial company and obviously a lot of his words were very salesy. Which is fine. But it 
literally is, rectangle in the room. That’s it. So, in many ways, it’s really, phenomenally, 
useful. And I’m very glad they’re doing it. I’m glad we have a bit of a connection to them 
now. But the whole issue of taking something out of the screen and putting it somewhere 
else, it was partly using their system that made me realize that’s not possible. And that’s 
actually kind of a big deal. So that’s that. And the meeting that Elliot and I had today, he 
mentioned who it was with. And I didn’t want to put too much into the record on that. But it 
was really interesting. The meeting was because of Visual-Meta. Elliot introduced us to these 
people. And Vint. Vint couldn’t be there today. We started a discussion. They have all kinds 
of issues with Visual-Meta. They love the idea, but then their implementation issue, blah, 
blah, blah. But towards the end, when I started talking about the Metaverse thing, they had no 
idea about the problems that we have learned. And they were really invigorated and stressed 
by it. So, I think what we’re doing here, in this community, is right on. I’m going to try now 
to rewrite some of the earlier stuff, to write a little piece over the weekend on academic 
documents in the Metaverse to highlight the issues. And if you guys want to contribute some 
issues to that document, that would be great or not, depending on how you feel. But I think 
they really understood that, what I said to them at the end is, if you have a physical meeting 
of a piece of paper, you can do whatever you want. But in the Metaverse, it can only do with 
the document, whatever the room allows you to, which is mind-blowingly crazy. And they 
represent a lot of really big publishers within medicine. They are under the National Institute 
of Health, as I understand. I’m not sure if Elliot is still in the room. So, yeah. It is good that 
we are looking in the right areas. 

Brandel Zachernuk: Yeah, that’s really constructive. For my part, one of the things that I’ve 
realized is that the hypertext people, the people who understand the value of things, like 
structured writing, and relationship linking, and things like that, are far better positioned than 
many, possibly most, to understand some of the questions and issues that are intrinsic to the 
idea of a Metaverse. I was watching, so I linked a podcast to some folks, it’s called, I think is 
it called Into The Metaverse, but it was a conversation between a VP of Unreal and the and 
the principal programmer, whatever, architect of Unity. So Vladimir Vuki¢evic, who was who 


created Unreal and Unity, and Vukicevic¢, I don’t know if I’m garbling that name, he was the 
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inventor of WebGL. Which is the foundation for all of the stuff that we do in virtual reality on 
web, as well as just being very good for being able to do fancy graphics, as I do at work and 
things like that. But their view of what goes into a Metaverse what needs to be known about 
entities relationships descriptions and things was just incredibly naive. I'll link the videos, but 
they see the idea of a browser as being intrinsic. And another person, who’s a 25-year veteran 
of Pixar and the inventor of the Universal Scene Description format, USD, which as you may 
know, Apple is interested in, sort of, promoting as being useful in the form of what this 
format of choice for augmented reality, quick look files, things like that. And again, just 
incredible naivete in terms of what are important things to be able to describe with regard to 
relationships, and constraints, and linkages of the kind that hypertext is. It’s the bread and 
butter of understanding how to make a hypertext relevant notionally and structurally, in a way 
that means that it’s (indistinct). So, yeah. It’s exciting, but it’s also distressing to see how 
much that thinking of people who are really titans of an interactive graphics field don’t know 


what this medium is. So, that looks fun. 


Frode Hegland: Yeah, it’s scary and fun. But I think we’re very lucky to have Bob here, 
because I’ve been very about the document and so on, and for about to say, “Well, actually, 
let’s use the wall as well”. It helps us think about going between spaces. And what I 
highlighted in the meeting earlier today was, what if I take one document from one 
repository, and let’s say, it has all the meta, so I’ve put a little bit here, a little bit there, but 
then, I have another document, from a different repository over here and I draw a connection 
between them. That connection now is a piece of information too. Where is stored? Who 
owns it? And how do I interact with that in the future? These are things that are not even 
begun to be addressed, because I think, all the companies doing the big stuff just want 
everything to go through their stuff. 

Bob Horn: And what kind is it? That is the connection. 

Frode Hegland: Yeah, exactly. So, we’re early naive days, so we need to produce some 
interesting worthwhile questions here. Fabien, I see your big yellow hand. 

Video: https://youtu.be/2NcS5COrVw24?t=4369 

Fabien Benetou: I’1I put the less yellow hand on the side. Earlier when I said, I don’t know 
what I’m doing, it wasn’t like fake modesty or trying to undermine my work or this kind of 
thing. I actually mean it. I do a bunch of stuff and some of the stuff I do, I hope is interesting. 
I hope is even new, and might lead to other things. But in practice, it’s not purely random, 
and there are some let’s say, not heuristic, but there are some design principles, philosophy 
behind it, understanding of some, hopefully, core principle of urology, or cognitive science, 
or just engineering. But in practice, I think we have to be humble enough about this being a 


new medium. And figuring it out is not trivial, it’s not easy, and it’s not, I think, it is part of it, 
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is intelligence and knowledge, but a lot of it is all that, plus luck, plus attempting. 

Frode Hegland: Oh, I agree with you. And I see that in this group, the reason I said it was I 
just wanted him to have a clue of the level of who we are in the room. That’s all. I think our 
ignorance in this room is great. I saw this graphic when I started studying, I haven’t been able 
to find the source, but it showed if you know this much about a subject, the circumference is 
the ignorance, it’s small. The more you know, the bigger circumference it is. And I found that 
to be such a graphic illustration of, you know something, you don’t know. We need to go all 
over the place. But at least we’re beginning to see some of the questions. And I think that’s a 
real contribution of what we’re doing here. So, we just got to keep on going. Also, as you 
know, we now have two presenters a month, which mean, for the next two or three months, 
I’ve only signed up one. Brandel is going to be doing, hopefully, in two to three weeks 
something, right? 

Brandel Zachernuk: Yeah. I’m still chipping away. Then I realized that there’s some reading 
I need to do, in order to make sure that I’m not mischaracterizing Descartes. 

Frode Hegland: Okay, that sounds like fun. Fabien, would you honour us, as well, with 
doing a hosted presentation over the next month or two or something? 

Fabien Benetou: Yeah, with pleasure. 

Frode Hegland: Fantastic! Our pathetic little journal is growing slightly less pathetic by the 
month. 

Fabien Benetou: | can give a teaser on... I don’t have a title yet, but let’s say, how a librarian, 
what a librarian would do if they were able to move walls around. 

Frode Hegland: That’s very interesting. It was good the one we had on Monday, with Jad. It 
was completely different from what we’re looking at. Looking at identity. And for you to now 
talk about that aspect, is kind of a spatial aspect, that’s very interesting. 

Bob Horn: I’m looking forward to whatever you write about this weekend, Frode. Because 
for me, the summaries of our discussions, with some organization, not anywhere near perfect 
organization, not asking for that, but some organization, some patterns are what are important 
to me. And when I find really good bunches of those, then I can visualize them. So, I’m still 
looking for some sort of expression of levels of where the problems are as we see it now. In 
other words, there were the, what I heard today, with Immersed, was a set of problems at a 
certain level, to some degree. And then, a little bit in the organization of knowledge, but not a 
lot, but that’s what came up in our discussion afterwards and so forth. So, whenever there’s 
that kind of summary, I really appreciate whatever you do in that regard, because I know it’s 
the hardest work at this stage. So I’m trying to say something encouraging, I guess. 


Frode Hegland: Yeah, thank you, Bob. That’s very nice. I just put a link on this document 
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that I wrote today. The next thing will be, as we discussed. But information has to be 
somewhere. It’s such an obvious thing, but it doesn’t seem to be acknowledged. Because in a 
virtual environment, we all know that you watch a Pixar animation, they’ve made every 
single pixel on the screen. There is no sky even. We know that. But when it becomes 


interactive, and we move things in and out. Oh, Brandel had a thing there. 


Brandel Zachernuk: One of the things that they that Guido Quaroni talks about, as as well 
as people have talked a bunch about, some of the influences and contributions of. Quilez 
makes Shadertoy, I don’t know if you’ve ever seen them or heard of that. But it’s this 
raymarched based fragment shader system for being able to do procedural systems. And so, 
none of the moss in brave, if you’ve seen that film, exists. Nobody modeled it. Nobody 
decided which pieces should go where. What they did was, Quilez has this amazing mind for 
a completely novel form of representation of data. It’s called the Signed Distance 

Fields raymarched shader. And so it’s all procedural. And all people had to do was navigate 
through this implicit virtual space to find the pieces that they wanted to stitch into the films. 
And so, it never existed. It’s something that was conjured on a procedural basis and then 
people navigated through it. So yes, things have to exist. But that’s not because people make 
it, sometimes. And sometimes it’s because people make a latent space, and then, they 
navigate it. And I think that the contrast between those two things is fascinating, in terms of 
what that means creative tools oblige us to be able to do. Anyway. 

Frode Hegland: Oh, yeah. Absolutely. Like No Man’s Sky and lots of interesting software 
out there. But it’s still not in the world, so to speak. One thing I still really want, and I’m 
going to pressure you guys every time, no, it’s not to write your bio, but it is some 
mechanism where, as an example, our journal, I can put it in a thing so that you guys can put 
it in your thing. Because then we can really start having real stuff that is our stuff. So if you 
can keep that in the back of your mind. Even if you can just spec how it should work, I'll try 
to find someone to do it, if it’s kind of rote work and not a big framework for you guys. 
Brandel Zachernuk: Yeah, | definitely intend to play more with actually representing text 
again. And somebody made a sort of invitation slash prompt blast challenge to get my text 
renderings to be better. Which means that I’1l need something to do it better on. And so, yeah. 
I think that would be a really interesting target goal. 

Frode Hegland: Awesome. Fabien, I see you have your hand, but on that same request to 
you guys, imagine we already have some web pages where you can click at the bottom, view 
in VR, when you’re in the environment. That’s nice. Imagine if we have documents like that, 
that’ ll be amazing. And I don’t know what that would mean, yet. There are some thoughts, 


but it goes towards the earlier. Okay, yes. Fabien, please? 


Fabien Benetou: Yeah, I think we need to go a bit beyond imagining. Then we can have 
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some sandbox, some prototypes of the documents. We have recorded, that’s how I started, the 
first time I joined, you mentioned Visual-Meta. And then, I put a PDF and some of the media 
data in there. No matter how the outcome was gonna exist, so I definitely think that’s one of 
the most interesting way to do it. The quick word on writing, my personal fear about writing 
is that, I don’t know if you know the concept, and I have the name of the people of my 
tongue, but yeah, ID Depth. So the idea is that you have too many ideas, and then at some 
point, if you don’t realize some of them, if you don’t build, implement, make it happen, 
however the form is, it’s just crushing. And then, let’s say, if I start to write, or prepare for the 
presentation I mentioned just 30 minutes or 10 minutes ago, the excitement and the problem 
is, it’s for sure, by summarizing it, stepping back, that’s going to bring new ideas. Like, “Oh, 
now I need to implement. Now I need to test it”. There is validation on it. I’m just not 
complaining or anything. Just showing a bit my perspective of my fear of writing. And also 
because in the past, at some point I did just write. I did not code anything. It felt good in a 
way. But then also. a lot of it was, I don’t want to say bullshit but, maybe not as interesting as 
that or it was maybe a little, so I’m just personally trying to find the right balance between 
summarizing, sharing, having a way that the content can be reused, regardless of the 
implementation, any implementation. Just sharing my perspective there. 

Frode Hegland: That is a very important perspective. And it is very important to share. And 
I think we’re all very different in this. And for this particular community, my job as, quote- 
unquote editor, is to try to create an environment where we’re comfortable with different 
levels. Like Adam, he will not write. Fine. I steal from Twitter, put it in the journal, and he 
approves it. Hopefully. Well, so far he has. So, if you want to write, write. But also, I really 
share, so strongly, the mental thing you talked about. We can’t know what it’s like to hear 
something until it exists. And we say, if an idea is important write it down, because writing it 
down, of course, helps clarifying. But that’s only if it’s that kind of an idea. Implementing, in 
demos and code is as important. I’ve been lucky enough to be involved with building our 
summer house, in Norway, doing a renovation here. And because it’s a physical environment, 
even doing it in SketchUp it’s not enough. I made many mistakes. Thankfully, there were 
experienced people who could help me see it in the real thing. Sometimes we had to put 
boards up in a room to see what it would feel like. So, yeah. Our imaginations are hugely 
constrained. So, it’s now 19 past. And Brandel was suggesting he had to go somewhere else. I 
think it’s okay, with a small group, if we finish half-past, considering this will be transcribed, 
anyway. And so, let’s have a good weekend. Unless someone wants a further topic 
discussion, which I’m totally happy with also. 

Brandel Zachernuk: Yeah. I’m looking forward to chatting on Monday. And I will read 
through what you sent to the group that you discussed things with today. Connecting to 
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people with problems that are more than graphical, and more than attends to the Metaverse, I 
think is really fascinating. Providing they have the imagination to be able to see that, what 
they are talking about is a “Docuverse”’. Is these sort of connected concepts that Bob has 
written about. I’ve got a book but it’s on the coffee table. The pages after 244. The 
characterization of the actual information and decision spaces that you have. It’s got the 
person with the HMD but then it’s sort of situated in an organization where there are flows of 
decisions. And I think that, recognizing that we can do work on that is fascinating. 

Bob Horn: I can send that to everybody, if you like. 

Frode Hegland: Oh, I have it. So without naming names or exactly who I was speaking to 
today since we’re still recording. The interesting thing is, of course, this feeds the, starting 
with the Visual-Meta, it feeds into some part of the organization desperately wants something 
like that and they’ve been pushing for years. But there are resources, and organization, and 
communication, all those real-world issues. So then, a huge problem is, I come in as an 
outsider and I say, “Hey, here’s a solution. It’s really cheap and simple”. It’s kind of like ’m 
stealing their thunder, right? I am not doing that, I’m just trying to help them realize what 
they already want to do. And today, when they talked about different standards, I said, “Look. 
Honestly, what’s in Visual-Meta, I don’t care. If you could, please, put it in BibTeX, the basic 
stuff, but if you want to have some json in there, it’s not something I would like, but if you 
want to do it there’s nothing wrong with that”. So, to try to make these people feel that they 
are being enabled, rather than someone kind of moving them along is emotionally, human 
difficult. And also, for them to feel that they’re doing something with Vint Cerf. All of that, 
hopefully, will help them feel a bit of excitement. But I also think that the incredibly hard 
issues with the Metaverse that we’re bringing up also unlock something in their imagination. 
Because, imagine if we, at the end of this year, we have a demo, where we have a printed 
document, and then we pretend to do OCR, we don’t need to do it live, right? And then, we 
have it on the computer, very nice. And now, suddenly, we put on a headset. You all know 
where I’m going with this, right? We have that thing. But then, as the crucial question you 
kept asking Gavin, and I’m glad you both asked it, Fabien and Brandel, what happens to the 
room when you leave it? What happens to the artifacts and the relationship if we solve some 
of that? What an incredibly strong demo that would be. And also, was it a little bit of a wake- 
up call for you guys to see that this well-funded new company is still dealing with only 
rectangles? 

Brandel Zachernuk: No. I know from my own internal experience just how coarse the 
thinking is, even with better funding. 

Frode Hegland: Yeah. And the greatest thing about our group is, we have zero funding. And 


we have zero bosses. All we have is our honesty, community, and passion. Now, it’s a very 
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different place to invent from. But look at all the great inventions. Vint was a graduate 
student, Tim Berners-Lee was trying to do something in a different lab. You know all the 
stories. Great innovations have to come from groups like this. I don’t know if we’re going to 
invent something. I don’t know. I don’t really care. But I really do care, desperately, that we 
contribute to the dialogue. 

Brandel Zachernuk: Yeah, | think that’s valuable. I think that the fact that we have your 
perspective on visual forms of important distilled information thought is going to be really 
valuable. And one of the things I’d like to do, given that you said that so many people make 
use of Vision 2050 is start with that as a sculpture, as a system to be able to jump into further 
detail. Do you have more on that one? 

Bob Horn: Well, I can take it apart. I can do what different things we want to do with it. For 
example, when we were clearing it with the team that worked that created some of the 
thought that went into it, the back cast thought, I would send the long trail of the four decades 
of transportation to Boeing, to Volkswagen, and to Toyota. I didn’t send it to the rest of the 
people. So, I could take that, I actually took that out and sent a PDF of that, only that to them. 
And that’s one dimension. Another dimension is that five years later, I worked on another 
project that was similar called Poll Free. Which is also on my website. And it narrowed the 
focus to Europe, to the European Union, rather than the whole world. But the structure is 
similar in many ways. So each one of those are extractable. Then also, I have a few... The 
two or three years after working on the Vision 2050, I would give lectures of different kinds. 
And people would ask me, “Well, how are we doing on this or that requirement?” And so, I 
would try to pull up whatever data there was, two, or three, or four years later, and put that in 
my slides, so there, that material is available. So, that we can extract, you could demo, at least 
that, “Here’s what we thought in 2010 and here’s what it looked like in 2014’. For one small 
chunk of the whole picture. So, yeah. And I have several, maybe I don’t know, six or eight, at 
least of those, that where I could find data easily and fast. So, there’s a bit of demo material 
there that one could portray a different kind of a landscape than the one that you were pointed 
out just a minute ago. 

Brandel Zachernuk: Yeah. That would be really interesting to play with. I was just looking 
to add some of the things. I think that the one thing that I had seen of the Vision 2050 was the 
fairly simple one, it’s a sort of a four, this node graph here, the nine billion people live well 
and within the limits of the planet I hadn’t seen yet. The sustainable pathway toward a 
sustainable 2050 document that you linked here on your site, which has a ton more 
information. And, yeah. One of the things that I’m curious about, one of the things that I 
think I will do to play with it first is actually get it into, not into a program that I write, but 
into a 3D modelling APP, to tear it apart, and think about the way in which we might be able 
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to create and distribute space for it. But first, do you have thoughts about what you would do 
if this was an entire room? It obviously needs to be a pretty big mural, but if it was an entire 
room, or an entire building, do you have a sense of the way in which it would differ? 

Bob Horn: Until you ask the question, and put it together with the pages from the old book, I 
haven’t really thought of that. But from many of the places in Vision 2050 one would have 
pathways like this. This was originally a pert chart way back when that I was visualizing, 
because I happened to have, early my career edited a book on pert charts for Dupont. And so, 
that’s a really intriguing question. To be extracting in and laying it out and then, connecting 
those and also flipping the big mural, the time-based mural in Vision 2050, making that flat, 
bringing different parts of it up, I think would be one of the first ways that one would try to 
explore that, because then, one could (indistinct) pathways, and alternatives, and then 
linkages. So, they’re different. Depending on one’s purpose, thinking purpose, one would do 
different things. 

Fabien Benetou: Brief note here. I believe, using Illustrator to make the visuals, I believe 
Illustrator can also save to SVG. And SVG then can be relatively easily extruded to transform 
a 2D shape into a 3D shape. Honestly, doing that would be probably interesting but very 
basic, or very naive. It’s still, I think, a good step to extrude part of the graph with different 
depth based on, I don’t know, colour, or meaning, or position, or something like this. So, I 
think it could be done. But, if you could export one of the poster in that format, in SVG, I 
think it would be fun to tinker with. But I think, at some point, you personally will have to 
consider, indeed, the question that Brandel asked. If you have a room, rather than a wall 
beyond the automatic extraction or extrusion, how would you design it? 

Brandel Zachernuk: Yeah. It’s something that I think would be really useful as an exercise, 
if you want to go through one of those murals and with a sketchbook, just pencils. And at 
some point, you can go through with us to characterize what I think, like you said, different 
shapes, different jobs call for different shapes through that space. But one can move space 
around, which is exciting. Librarians can move their walls around. 

Bob Horn: | was going to say the other, if you strike another core, just as from the 
demonstration we saw earlier this morning. The big mural could be on one wall. There was a 
written report. There is a 60 or 80-page report that could be linked in various ways to it. And 
it exists. And then, there’s also, in that report, there’s a simplification of the big mural. It 
reduces the 800 steps in the mural to about 40. And it’s a visual table look. So, already there 
are three views, three walls, and we’ve already imagined putting it flat on the floor and things 
popping up from it. All right, there we go. There’s a room for you. 

Brandel Zachernuk: Exciting, yeah. I think that’s a really good start. And from my 
perspective, I think that’s something that I can and will play with is, starting from that JPEG 
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of the PDF, I’ll peel pieces of that off and try to arrange them in space, thinking about some 
of the stuff that Fabien’s done with the Visual-Meta, virtual Visual-Meta. As well as what 
Adam succeeded in doing, in terms of pulling the dates off, because I think that there’s some 
really interesting duality of views, like multiplicity of representations that we can kind of get 
into, as well as being able to leverage the idea of having vastly different scales. When you 
have a, at Apple we call it a type matrix, but just the texts and what what’s a heading what’s a 
subhead. But the thing is that, except in the most egregious cases, which we sometimes do at 
Apple, the biggest text is no more than about five times the smallest text. But in real space 
you can have a museum, and the letters on the museum wall or in a big room are this big. 
And then you have little blocks like that thing. And there’s no expectation for there to be 
mutually intelligible. There’s no way you can read this, while you’re reading that. But 
because of the fact that we have the ability to navigate that space, we can make use of those 
incredibly disparate scales. And I think that’s incumbent on us to reimagine what we would 
do with those vastly different scales that we have available, as a result of being able to 
locomote through a virtual space. 

Bob Horn: Well, let me know if you need any of these things. I can provide, somehow. I 
guess you and I could figure out how to do a dropbox for Illustrator or any other thing that 
can be useful for you. 

Brandel Zachernuk: Yeah, thank you. I may ask for the Illustrator document. One of the 
things that I’ve been recently inspired by, so there’s an incredible team at Apple that I’m 
trying to apply for called prototyping. And one of the neat things that they have done over the 
years is describe their prototypic process. And it mostly involves cutting JPEGs apart and 
throwing them into the roughest thing possible in order to be able to answer the coarsest 
questions possible first. And so, I’m very much looking forward to doing something coarse 
ground with the expectation that we have a better sense of what it is we would want to do 
with more high fidelity resources. So, hopefully that will bear fruit and nobody should be, 
hopefully not, too distraught by misuse of the material. But I very much enjoy the idea of 
taking a fairly rough hand to these broad questions at first, and then, making sure that 
refinement is based on actual resolution, in the sense of being resolved, rather than pixel 
density. 

Bob Horn: Yeah, well, okay. If you want JPEGs we can make JPEGs too. 

Frode Hegland: You said almost as a throwaway thing there. Traverse. But one thing that I 
learned, Brandel, particularly with your first mural of Bob’s work is that, traversal, unless 
you're physically walking if you have room scale opportunity, is horrible. But being able to 
pull and push is wonderful. And I think that kind of insight that we’re learning by doing is 


something we really should try to record. So, I’m not trying to push you into an article. But if 
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you have a few bullets that you want to put into Twitter, or sent to me, or whatever, as in, this, 
in your experience has caused stomach pain, this hasn’t. Because also, yesterday, I saw a... 
You know I come from a visual background, and have photography friends, and do videos, 
and all that stuff, suddenly, a friend of mine, Keith, from some of you have met, we were in 
SoHo, where he put a 8k 360 camera, and it was really fun. So, I got all excited, went home, 
looked up a few things, and then I found the Stereo 180 cameras. And I finally found a way to 
view it on the Oculus. It was a bit clunky, but I did. It was an awful experience. There’s 
something about where you place your eye. When we saw the movie, Avatar, it was really 
weird that the bit that is blurry would actually be sharp as well, but somewhere else. Those 
kinds of effects. So, to have a stereoscopic, if it isn’t exactly right on both eyes and you’re 
looking at the exact, it’s horrible. So, these are the things we’re learning. And if we could put 
it into a more listy way, that would be great. Anyway, just since you mentioned. 

Brandel Zachernuk: Yes. It’s fascinating. And that’s something that Mark Anderson also 
observed when he realized that, unfortunately, the Fresnel lenses that we make use of in 
current generation hardware means that, it’s not particularly amenable to looking with your 
eyes like that. You really have to be looking through the center of your headset in order to be 
able to get the best view. You have this sense of the periphery. But will tire anybody who tries 
to read stuff down there, because their eyes are going to start hurting. 

Frode Hegland: Yeah. I still have problems getting a real good sharp focus. Jiggle this, 
jiggle that. But, hey! Early days, right? So when it comes to what we’re talking about with 
Bob’s mural, and the levels, and the connections, and all of that good stuff, it seems to be an 
incredibly useful thing to experiment with exactly these issues. What does it actually mean to 
explode it, et cetera? So, yeah. Very good. 

Fabien Benetou: Yeah. I imagine that being shared before. But just in case, Mike Elgier, 
who is, or at least who was, I’m not sure right now, but a typist and designer at Google, on 
the UXL product. Wrote some design principle a couple of years ago. And not all of these 
were his, but he illustrated it quite nicely. So, I think it’s a good summary. 

Brandel Zachernuk: Yes, I agree. He’s still at Google he was working on Earth and 
YouTube. Working on how to present media, and make sure that it works seamlessly so that 
you're not lying about what the media is, but in terms of presenting a YouTube video in VR 
in a way that it isn’t with no applied and like I see it screen or whatever. But also, making 
sure that it’s something that you can interact with as seamlessly as possible. So, it’s nice 
work, and hopefully, if Google ramps up its work back into AR, VR, then they can leverage 
his abilities. Because they’ve lost a lot of people who are doing really interesting things. I 
don’t know if you saw, Don McCarthy has now moved to New York Times to work on 3D 


stuff there. And that’s very exciting for them. But a huge blow for Google not to have them 


298 


back. 
Frode Hegland: Just adding this to our little news thing. Right. Excellent. Yeah. Let’s 


reconvene on Monday. This is good. And, yeah. That’s all just wonderful. Have a good 
weekend. 
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Chat Log 


16:46:14 From Fabien Benetou : my DIY keyboard passthrough in Hubs ;) 
https://twitter.com/utopiah/status/125012 1506782355456 

using my webcam desktop 

16:48:25 From Frode Hegland : Cool Fabien 

16:50:49 From alanlaidlaw : that’s the right call. APIs are very dangerous in highly dynamic 


domains 
16:51:47 From Fabien Benetou : also recent demo on managing screens in Hubs 


https://twitter.com/utopiah/status/1493315471252283398 including capturing images to move 


them around while streaming content 


17:03:43 From Fabien Benetou : good point, the limits of the natural metaphor, unable to get 


9999. 


the same affordances one does have with “just’”’paper 

17:04:07 From Frode Hegland : Carmack? 

17:04:16 From Frode Hegland : Oh that was Quake 

17:04:48 From Frode Hegland : Can you put the names here in chat as well please? 
17:05:16 From Fabien Benetou : Vladimir Vukicevié tire 

17:05:53 From Frode Hegland : Thanks 

17:06:40 From Brandel Zachernuk : This is Vuki¢evic¢: 
https://cesium.com/open-metaverse-podcast/3d-on-the-web/ 

17:07:17 From Brandel Zachernuk : And Pixar/Adobe, Guido Quaroni: 
https://cestum.com/open-metaverse-podcast/the-genesis-of-usd/ 

17:11:09 From Frode Hegland : From today to the NIH: 
https://www.dropbox.com/s/9xyl6xgmaltojqn/metadata%20in%20crisis.pdf?dl=0 
17:11:25 From Frode Hegland : Next will be on academic documents in VR 
17:12:07 From Fabien Benetou : very basic but the documents used in 
https://twitter.com/utopiah/status/1243495288289050624 are academic papers 
17:13:19 From Frode Hegland : Fabien, make an article on that tweet?... 
17:13:30 From Fabien Benetou : length? deadline? 

17:13:34 From Frode Hegland : any 

17:13:44 From Frode Hegland : However, do not over work! 

17:13:54 From Frode Hegland : Simple but don’t waste time editing down 


17:14:07 From Fabien Benetou : sure, will do 
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17:14:11 From Frode Hegland : Wonderful 

17:14:52 From Fabien Benetou : (off topic but I can recommend 
https://podcasts.apple.com/be/podcast/burnout-and-how-to-avoid-it/id1474245040? 
i=1000551538495 

on burn out) 

17:28:05 From Brandel Zachernuk : 
https://www.bobhorn.us/assets/sus-Suc-vision-2050-wbcsd-2010-(1).pdf 
17:28:17 From Brandel Zachernuk : 
https://www.bobhorn.us/assets/sus-6uc-pathwayswbcsd-final-2010.jpg 

17:39:10 From Fabien Benetou : https://www.mikealger.com/ 

17:39:27 From Fabien Benetou : design principles for UX in XR, pretty popular 
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Harold Thimbleby 


Getting mixed text right is the future of text 


When we read text, at least text that we are enjoying as we read it, we get immersed in it, and 
it becomes like a stream of consciousness we willingly join in with. We lose awareness of the 
magic reading skills that took us years to learn — these marks on screen or paper somehow 
create mental images or sounds, feelings like laughter, disagreement, anger, plans for action, 
anything, in our heads. If we pause from the flow, we may reflect about the text’s metadata 
— who wrote this; when did they write it; how much do we have to pay for it; when was it 
written? — we want to know lots details about the text. 

If we are feeling critical, we may notice the typography: some text is italic, the page 
numbers are in a different font, there are rivers in the paragraphs, and the kerning perhaps 
leaves a lot to be desired. Then we notice how the author italicises Latin phrases, like ad 


nauseam, but does not italicise Latin abbreviations like e.g. for example. 


If we are programmers, we might wonder how the text works, how it was actually 
implemented. What is the data format? How did the writer and the developers store this 
information, and yet convey a coherent stream of consciousness to the readers? Some texts 
mix in computed texts, like indices and tables of contents; then there are footnotes, side 
notes, cross references, running headings, page numbers — all conventional ways of mixing 
in different types of text to help the reader. 

If the text is on a web page or represented in VR, even more will be happening. VR text 
is typically interactive. Perhaps it scrolls and pans in interesting ways, is reactive to different 
sorts of reading devices, fitting into different screen sizes and colour gamuts, and it probably 
interactively needs information from the reader. Increasingly, the reader will need to 
subscribe to the text, and the details of that are held in very complex metadata stored in the 


cloud, far away from the text itself yet linked back to it so the reader can have access to it. 
The author's experience of text 


For the sake of concreteness, familiarity, and simplicity, we will use HTML as an initial case 
study. 


HTML is a familiar, well-defined notation, and it is powerful enough to represent almost 


any form of text. For example, Microsoft Word — which provides a WYSIWYG experience 
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for the author — could easily represent all of its text using HTML; in fact, Word now uses a 
version of XML (which is basically a fussy version of HTML) to do so. Furthermore, in this 
chapter it’s helpful that we can talk about HTML on the two-dimensional printed (or PDF or 
screen) page, unlike examples from VR. (If we had used Microsoft Word as the running 
example, it has plenty of mixed texts, like tables of contents, references, forms. Even basic 


features like tables and lists are very different sorts of text than the main document text.) 


Despite the widespread use of HTML across the web, and its widespread use in highly 
critical applications, such as managing bank accounts and healthcare services and writing 
pilot operating manuals for aircraft, HTML is a surprisingly quirky and unreliable language 
for text. The main reason for its quirkiness is that HTML was originally designed to 
implement some innovative ideas about distributed hypertext, and nobody then thought it 
would develop to need designing to be safe to use in critical applications, let alone that it 


would need designing to integrate reliably with many other notations. 


We’ ll give some examples. If you get bored with the details, do skip forward to the end 


of this chapter to see what needs to be learned to improve future mixed text. 


Remember these examples illustrate problems that can occur when any text mixes any 
notations, but using HTML makes it easy to describe. (Also, you can easily play with my 
examples in any web browser.) We’ll take very simple examples of mixed text, not least to 
wonder why even simple mixes don’t work perfectly. For brevity, we’ll ignore the 
complexities and flaws of mixed texts like tables of contents, indices, and so on (there aren’t 
many word processors that ensure even just the table of contents has the right page numbers 
all the time). 

In addition to the text, styles and layout HTML can define, HTML allows developers to 
mix comments in the text. Comments are texts that are intended to be read by developers but 
not seen by readers. Perhaps a developer is in a hurry for people to read a text but they 
haven’t yet completely finished it. How will the developer keep track of what they want to 
write but haven’t yet done? One easy solution is to use a comment: the developer writes a 
comment like “XX I need to finish writing this section by December” or “I need to check 
this! What’s the citation?” or “I must add the URL later’, but the readers of the text won’t see 
these private comments. The developer, as here, might use a code like XX so that they can 
easily use search facilities to find their important comments where they need to do more 
work. 

The actual notation for comment in HTML is <!-- comment -->. Here, I’ve used another 
mixture of texts: the italic typewriter font word comment (in the previous sentence) is being 
used to mean any text that is used as comment and hence will not be visible to the text’s 


reader. 
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One problem with this HTML notation is that it is not possible to comment out arbitrary 
HTML: if it already contains comments, where the commented out HTML will end with the 


first -->, not with the last. 


Why would you want to comment out entire blocks of HTML, which might contain 
further comments? A very common reason to do this is that the HTML text is not working 
properly: there is some sort of bug in the text. One of the fastest ways of finding the cause of 
the problem is to systematically comment out chunks of the text. If commenting out this bit 
doesn’t affect the bug, the bug must be somewhere else. Try again, and continue doing this 
until the bug is precisely located. (There are systematic ways to do this that speed up the 
debugging, like binary search.) 

HTML is structured using tags. A simple tag is <p>, which generally starts a paragraph. 
Tags can also have parameters (HTML calls them attributes) to provide more specific control 
over their meaning or features. For example, <p title = "This paragraph is about HTML"> 
typically makes the specified title text appear when the user mouses over the paragraph. The 
spaces in this title mean that it has to be written between two quote symbols (the two " 
characters) — otherwise the four words here after the first, paragraph, is, about and HTML, 
would be taken as further attributes; the title would just be set to This, and all the other words 
would be si/ent errors. However, we obviously want the entire text to be a single value made 
up of all the words and spaces between them. Unfortunately what is obvious to us is not 
obvious to HTML. HTML has to cope with many authors’ ideas that are not obvious, most of 
which won’t be so obvious to us, so it needs another feature to avoid it having to somehow 
intuit what we think we mean. So, sometimes, but not always, we have to use " around 


attribute values. 


Unfortunately, using " around attribute values means that yet another random convention 


is needed if we need " itself to be part of a value. 
For example, 
<h1 title = "This is the beginning of the book "The Hobbit""> 


does not work. Instead, the HTML author is required to use a single quote instead. Here, this 


would do: 
<h1 title = 'This is the beginning of the book "The Hobbit"'> 


— which solves that problem, but now we are in a mess if for any reason we need both sorts 
of quote. So, what about the title of a book about a book? 


<h1 title = "J. R. R. Tolkien's "The Hobbit""> 


which needs to use both " and' in the attribute value! HTML cannot do that, at least without 


relying on even more conventions: for instance, knowing that any character in HTML can be 
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written as &code; we could correctly but tediously write 

<hl title="J. R. R. Tolkien&#39;s &#34;The Hobbit&#34;"> 
This is just bonkers isn&#39;t it? It relies on the author knowing what numeric codes (or 
names) need to be used for the problematic characters, and also relies on the author testing 
that it works. 

Other languages use a different, much better, system to allow authors to mix types of 
text. For instance in the widely-used programming language C, within a value like "stuff", 
characters can be represented by themselves, or more generally codes, after a slash. Thus \' 
means ', \ means ", and more generally \nnn means the character with code nnn like HTML’s 
own &#nn; but using octal rather than decimal. This approach means in C one could write a 
value for a book title like 

title = "J. R. R. Tolkien\'s \"The Hobbit\""; 
and it would work as intended — and it is much easier for the author to read and write. Note 
that the \' is being used correctly even though in this case a bare ' alone, without a slash, 
would have been equally acceptable too. So one must ask: given this nicer design of C, and 
nicer design or lots of similar, popular, textual languages which pre-dated HTML, why did 
HTML use a scheme that is so awkward? 

Note that a scheme like HTML’s that is sometimes rather than always awkward means 
that authors are rarely familiar with the rare problems. The problems come as surprises. 

HTML gets worse. 

HTML has ways to introduce further types of text, such as CSS, SVG, MathML, and 
JavaScript. For example, <script> document.write(27*39); </script> is JavaScript mixed 
inside of HTML text. Here the JavaScript is being used to work out a sum (namely, 27 times 
39) that the author found easier to write down in JavaScript than work out in their head. 
Moreover, JavaScript is often used inside HTML to generate CSS and SVG and other 
languages (such as SQL, which we will return to below). 

What an author can write in JavaScript has many very unusual constraints. 

Consider this simple example: 

<script> var endScript = "</script>"; </script> 
This will not work, because HTML finishes the JavaScript prematurely at the first </script> 
rather than the second one. HTML does not recognise JavaScript’s syntax, so it has no idea 
that the first </script> is inside a string in JavaScript and was not intended to be HTML at 
that moment, which the second one was. 
The workaround for this is a bit bizarre: HTML’s & entities can be used to disguise the <> 


characters from HTML! Here’s how it can be done: 
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<script> var endScript = "&lt;/script&gt;"; </script> 
I think we get so used to this sort of workaround, we lose sight of how odd it is to have to 
understand how two languages, here HTML and JavaScript, mess each other up before we 
can safely use either of them 

Here, next, is some routine JavaScript that displays an alert for the developer if (in this 
case) x>y, which might mean something has gone wrong: 

<script> if( x > y ) alert("--> x > y"); </script> 
Assume the author, or another author working on the same text, decided to comment out a 
stretch of HTML for some reason. Weirdly, this JavaScript will now produce the text “x > 
y"); -->”, because the ‘harmless’ arrow in the JavaScript code has turned into HTML’s --> 
end of comment symbol, even though it is still inside JavaScript. Confusingly, the JavaScript 
used to work before it was commented out! 

Ironically, because HTML is designed to ignore errors, when it is mixed with JavaScript, 
as here, authors may make serious errors (much worse than this simple example) that are 
ignored and which nothing helps them detect. In complex projects, especially with multiple 
authors sharing the same texts, such errors are soon impossible to avoid, and are very hard to 
track down and fix because they are caused by strange interactions between incompatible text 
notations. They aren’t errors in HTML; they aren’t errors in JavaScript; they are errors that 
only arise inside JavaScript inside HTML text. 

Here’s another confusion. Like HTML, JavaScript itself has comments. Thus, in 
Javascript, anything written after // to the end of the line is ignored. But // </script> is a 
JavaScript comment ignored by JavaScript but includes valid HTML that is not ignored by 
HTML. 


To summarise so far: HTML is a text notation that allows, indeed encourages and relies 
on, other languages (such as JavaScript) being mixed in, but HTML and these languages were 
developed independently, and they interact in weird and unexpected ways that can catch 
authors and readers out. 

These examples, chosen to be quick and easy to explain, may give the misleading 
impression that the problems are trivial. They may also, wrongly, give the impression that 
mixed text problems are restricted to HTML. But it gets worse. 

An HTML text may use JavaScript that needs to use the language SQL, a popular 
database language. The problem is that when SQL is embedded in JavaScript in HTML, it 
raises security risks. “SQL injection” is the most familiar problem. 

A user using an HTML text on a web page may be asked to enter some text, like some 


product they want to buy. The product needs to be found in the store’s database, so SQL is 


307 


used to make the connection. But if, instead of a product description, they type a bit of valid 
SQL, this SQL will go straight to the SQL engine. This is the SQL injection, and then the user 
(presumably a hacker) can get the SQL backend to do bad things. 

If a web site allows (by accident and ignorance) SQL injection, a hacker can do much 
damage by taking over and programming the SQL database. In addition to this problem, SQL 
has its own different weird rules for strings and mixing texts, making examples like the 
simple HTML+JavaScript problems look simple. To make matters worse, an SQL database 
may well store HTML and JavaScript, for instance to make nice descriptions of the products 
the store sells. So mixed text can mix text. 

Hackers can have fun with the bugs. There was a UK company registered under the name 
DROP TABLE "COMPANIES";—LTD, a company name that is contrived to be valid SQL. 
If injected into a database with a table called companies it would drop (that is, delete) the 


company’s data. 


Interesting aside... 


We’ve mentioned comments, and shown how they can be useful for authors of texts. HTML 
also allows text to be optionally hidden or made visible to readers, a sort of generalisation of 
comments but available to both authors and readers. This feature is the hidden attribute. Thus 
<span>Hello</span> says hello, but <span hidden>Hello</span> says nothing at all for the 
reader, a Jittle bit like <!-- hello --> would too. Ironically, to do anything useful, like allowing 
text — maybe an error message — to appear only when it is needed requires using JavaScript 


to dynamically edit HTML attributes (here, to interactively disable or enable hidden). 


Mixed texts in single systems 


Instead of mixing two text systems, like HTML and JavaScript, it ought to be easier to use a 
single integrated system. I’ve already hinted that there is more to the mixing of single-system 
texts like mixing in tables of contents into documents, but let’s stick with “trivial” mixing — 
because even that goes awry (and its weirdness is easier to explain briefly). 

I wrote this chapter using Microsoft Word. For the examples in HTML, I copied and 
pasted the text in and out of this chapter into a web browser, ran the text, and double-checked 
it did what I said it did. As I improved my discussion of the examples, text went backwards 
and forwards — hopefully without introducing errors or dropping off details, like the last > 


character in a bodged cut-and-paste. It would have been easier and more reliable had I used 
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an integrated mixed text system like Mathematica, then the entire text could have been 


authored in one place and could have stayed in place without any cut-and-pastes. 


In HTML if I say “<hr> is a horizontal rule,” then I have already used up the four letters 
<hr> to display themselves, namely as <, h, r, and >. (The fact that I actually had to write 
&lt;hr&gt; is another HTML mixed text problem.) In HTML I can’t reuse the same text to 
show what this <hr> does. However since Mathematica is programmable, I can write <hr> 
once and get it displayed numerous times, and each time processed in any way I like: 
sometimes to see the specific characters, sometimes to see how it renders (for instance as it 
would in HTML, as a horizontal rule), and sometimes to do arbitrary things. How many 
characters is it? 4. And if I changed the <hr> to, say, <hr style = "width: 50%; height: 1cm">, 
that 4 would change to the correct value of 38 without me doing anything. 

While Mathematica is an example of a sophisticated system originally designed for 
mixing text with mathematics, it still has text-mixing design flaws. For example, a 
Mathematica feature for embedding text inside text — exactly what this chapter is about — is 
called a string template in its terminology. String templates use the notation <* ... *> to 
indicate a place to mix arbitrary Mathematica text into strings of otherwise ordinary text, 
using <* ... *>a bit like HTML’s own <script> ... </script> notation. 

For example, here is a single line easily written in Mathematica: 


“The value of 2 is <* N[4ArcTan[1]] *>” turns into “The value of z is 3.14159” 


Very nice, but how would you write a string template that explains how to insert 
Mathematica text? You’d want to do this because using string templates to explain string 
templates would ensure the explanations were exactly correct. Indeed, Mathematica comes 
with a comprehensive user manual written as a Mathematica text, which does exactly this to 
illustrate how all its features work. Unfortunately, you can’t document string templates so 
easily (without complex and arbitrary workarounds). If I had written the example above 
entirely in Mathematica, the first <*, which you are supposed to read as showing how to use 
the mixed text feature, would already have been expanded, so the example wouldn’t work at 
all. “The value of z is 3.14159” turns into “The value of 2 is 3.14159” doesn’t say anything 
helpful! 

Mathematica allows you to write special characters from other texts explicitly. Thus the 
Greek (or Unicode) symbol \[pi] written in ordinary text can be used to mean z itself. If they 
had thought of having \[Less], which they don’t, then the <* problem would have been fixed. 
Yet they have LessEqual, for <, and lots more symbols. The omissions, like having no 
abbreviation Less, are arbitrary, even when they are needed, because Mathematica itself made 
<a special character! The designers of systems like HTML and Mathematica don’t seem to 


realise that a simple feature needs checking off for compatibility right across the language — 
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when string templates were introduced in Version 10.0 of Mathematica, evidently nobody 


thought to go back over the basic text notations introduced in Version 1. 


There are various workarounds of course, which perhaps experienced Mathematica 
users will be shouting at me. Ordinarily, though, an author of a text won’t realise 
workarounds are needed until after something unexpected goes wrong, then they have to 
waste time trying to find the problem, then find an ad hoc solution using tricks they have to 
work out for themselves. Remember, “experienced” authors are just those who have already 
come across and overcome these “trivial” problems. String templates are clever, but suddenly 
what was supposed to be empowering mixed text feature has turned into a slippery, wiggling 
eel 

We should not admire experienced authors who know all the problems and 
workarounds for mixed text. We should be despairing at the people who design mixed 


systems that don’t work reliably together. 


Future text mixed with Al and ... 


This chapter has discussed the unavoidable need for interleaved mixed text, so text can fulfill 
its many purposes — whether for authors or readers. It showed (mostly by way of HTML- 
based examples) that many practical problems remain. Mixing text leverages enormous 
versatility, but at the cost of complexity. The devil is in the details. 

We hinted that embedded languages like JavaScript can be used to help the author add 
power and features to text to enrich the readers’ experience. The example we gave was 
simple, but made the point: if the author does not know what 27 times 39 is, they can get 
JavaScript to work it out and insert the answer. Another example would be to display the date 
— JavaScript knows that even if the author doesn’t. These are simple examples of mixed text 
that build on computational features. 

The world of computation is rapidly expanding in scope and impact with new tools. 
Examples that can transform the author’s experience of writing include such AI tools as 

https://www.gomoonbeam.com 

https://elicit.org 

https://lex.page, and more. 

These fascinating AI tools can do research, can do writing, and can inspire people out of 
writer’s block. There are surprisingly many such tools, leveraging every gap imaginable in 
the writing and reading process. We are still learning how AI can help, and every way it helps 


relies on mixing in more forms of text together — they didn’t mix, then they would not be 
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contributing directly to the text or the author’s work. 


A final example is the use of programmable systems like Mathematica and R, which 
can mix text and computation and AI, as well as access curated databases of all manner of 
sources that can help the author. Unlike normal AI systems that are generally packaged up to 
do one thing well, Mathematica and R can be programmed by the author to help in any way. 

Mathematica, for instance, not only includes AI and ML and lots more, but can draw a 
map of Africa, get the country names and boundaries right and up to date, and find out all 
other details, like the weather in Sudan, its GDP or its adult literacy, even for very the day the 
reader reads about it, and mix it all in to the text the author is writing. Indeed, research papers 
often require detailed computations, often involving statistics, and doing this reliably mixed 
in the text, as Mathematica can, makes the papers much more reliable than when the 
computations being done conventionally — that is, done elsewhere and manually copied-and- 
pasted into the text, often introducing typos and other errors, as well as raising problems of 
the author forgetting to update the statistics when something relevant in the paper is updated. 


Consistency is a problem best solved by computers doing the text mixing. 
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Conclusions 


The future of text requires and cannot avoid mixing different sorts of text. We already 
interleave all sorts of text without thinking and often without problems. Occasionally, 
however, things get tricky. When we use internet technologies to leverage our mixed texts, 
they can be read and used by millions of people. This means that what seem like arcane tricky 
things to us and of no real importance can happen to hundreds or thousands of people, and 
can have dire consequences for them. 

Unfortunately, mixing different types of text is a mess. Text has become very powerful 
thanks to computers and computation; but text has also become unreliable thanks to the poor 
design and inconsistencies between different types of text. We gave examples of the mess of 
HTML and JavaScript being mixed, and examples of mixed text problems within the single 
Mathematica application. 

Developers keep adding new types of text to representations, historically HTML being 
a notable example, that were never intended to be extended so far as they have been. And 
each new type of text (CSS, MathML, etc) has to work with other and all previous types of 
text that did not anticipate it — to say nothing of the complexities of backwards compatibility 
with earlier versions of each type of text. The Catch-22 of “improving” the design of text 
often means compromising lots of text authored before the design was improved. 

Special cases routinely fail, and workarounds are complex and fragile. In a saner world, 
HTML, JavaScript, SQL, and all the other languages would have been designed to work 
closely and better together, with no need for author workarounds. 

It’s maybe too late to start again, but here are a few ideas that may help: 

- Authors should use checking systems, and use servers that check for known problems 
(like SQL injection). I’ve suggested that the standards for languages like HTML are 
inadequate, but at least checking that your text conforms to relevant standards is a start. 
Like spell-checking, it won’t fix all your problems, but it’s still really worth doing. 

- When new forms of text are invented, ensure they work well with existing types of text — 
in particular, by reporting errors so that authors do not release unreliable texts to 
unsuspecting readers. An extreme form of this idea is polyglot markup, which is markup 
(like HTML is) but designed to work in different dialects consistently. 

- Ifyou are a developer, and you find yourself writing very specific code like this: ... “<*” 

. ">". (Le., using <* and *> as built-in strings, as there must be somewhere inside 
the Mathematica implementation code) please notice those are totally arbitrary strings you 


devised, and there is no reason why the author — who is not you — using your system will 
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want exactly those codes. At least make them parameterisable so the author can work 
around clashes you failed to anticipate, or devise other ways to be more flexible. 

- Read up on other people’s attempts. For example, the reasoning behind the divergence 
between the different philosophies of HTML, particularly the snapshot based standards of 
HTML 4, 5, 5.1, etc, and the /iving standards most of us now user that are continually 
updating, is both fascinating and a warning. 

This chapter discussed a problem that is more generally called feature interaction. That is, 

texts have features, but in mixed texts the otherwise desirable features of each text interact in 

unhelpful and unexpected ways. In general, there are no good solutions to feature interaction, 
other than taking care to avoid it in the first place and providing mechanisms to help detect it 

(even block it) before any downstream reader is confused. In healthcare, the problem would 

be seen as a failure of the problem called interoperability, a potentially lethal problem that 

undermines the reliability of the mixed texts of patient records. 
If we are going to have feature interaction, which we are, we should take all steps to 
minimise it, and design the amazing powerful things mixed texts can do to eclipse their 


problems. 


http://www.harold.thimbleby.net 
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Jamie Joyce 


Guest Presentation : The Society Library 


https://youtu.be/PucSvzwp8IQ 


In case any of you don’t know who we are, we are The Society Library. 

We’re non-profit, a collective intelligence non-profit. I’m going to start this 
presentation just by talking about who we are and what we do. Then I’m actually going to 
show you what we're up to. And I’d love to store some of your feedback and some of your 
ideas because some of you have been thinking about these types of projects for decades, and 
I’m only three years in. I have been thinking about it for about seven to ten, but I’m only 
three years in, in terms of implementing these things. So I’d love to get feedback and to hear 


how you think we could grow and expand what we do. 
We’re The Society Library, the main projects that we’re working on are essentially: 


¢ How can we model societal scale deliberation? 

* How, in modelling societal scale deliberation, we can actually start creating ways in order 
to have more informed, inclusive, and unbiased decision-making? 

* How can we generate policy more collaboratively by taking in the inputs of individuals? 
And then, ultimately: 

¢ How can we have a common knowledge library on complex social and political issues that 


can inform the general public? 


All of these are very like pro-social, pro-democratic projects. The Society Library itself is 
attempting to fill a role in society. In the United States, we have something called the 
Congressional Revenue Service, it’s run by the Library of Congress. In the U.S., the 
Library of Congress actually prepares all of these lovely briefs and does all of this research 
for our members of Congress, our senators, and our house representatives. However, if I’m 
not mistaken, at the state and local level, as well as for the general public, these types of 
research services don’t exist. So our congresspeople, they can make a request to be brought 
up to speed on the topic of AGI and an entire library will work to organise and do all this 
research to deliver the knowledge products. And the Society Library is looking to do that for 
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the general public, and also at the local and state level in government. So some of these 
projects that we work on are deliberation mapping projects, like our great American debate 
program, which Ill talk about, specifically. We want to get into a project which we’ll 
probably rename because everyone hates it. I called it The Internet Government, people 
assume we’ve been governing the internet. We certainly do not mean that. It just means 
enabling governance platforms on the internet. So how can we generate policy? How can we 
produce decision-making models that are informed from the collective input and deliberation 
of? Essentially, what we’re aiming for is an entire nation. So we’re really working at the 
societal scale and I will talk about how we do that. And then, ultimately in our timeline, we 
want to fill this role as being accumulative common knowledge resource for the public. And 
so I’m going to talk about where we’re at right now, which is model and societal scale 


deliberations. And I’m going to get into how we actually go about accomplishing that. 


Currently, the topics that we’ve worked on, that we’ve cut our teeth on, in the past 
few years have been the topic of nuclear energy, climate change, COVID-19, and election 
integrity issues. We’ve mapped a few other spaces, as well, essentially, associating actors 
with actions in political movements. So, for example, we were mapping these kinds of things 
out in the George Floyd protest. But what we’re really interested in is finding what are the 
fundamental questions that society has about specific issues. Currently, we’re working within 
the English-speaking United States. And then, can we go ahead and deconstruct the collective 
knowledge content that we have, to go about compiling answers to those questions so that we 
can compare, and contrast what propositions, what positions, what arguments have more or 
less evidence. What kind of evidence, how much evidence, produced at what time in 
corroboration with what. We’re really interested in just being able to start visualising the 
complexity of our social and political discourse. So that, again, it can, down that timeline, 
start forming decision-making models in the production of policy. 

If it’s not obvious, to me what we discovered when we undertook the project of, Okay, 
let’s start mapping these debates is that, debates in the United States, on these high-impact 
persistent polarising issues, are actually unbelievably large. So the topic of climate change, I 
think we’re now up to 278 unique sub-topics of debate, and there can be tens of thousands of 
arguments and pieces of evidence in each one of those sub-topics. But interestingly, what we 
found is that all of these sub-topics correspond to answering only one of six questions. So all 
of these debates that are happening across various subjects related to conceptualising the 
problem of climate change and its severity, to solutions, and things like that, all of them are 


really responding to six fundamental questions. 


Our latest subject that we’re working on is nuclear energy, and we’re still assessing 


what those fundamental questions are. For COVID-19, because it was such a new subject, 
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and I think it was so global and so viral, we found over 500 and basically 13 fundamental 


questions. And for election integrity, there were 81 subtopics and two questions. 


Today I’m also going to be showing you a project that we’re going to be releasing at the 
end of this month, which I’m really excited about. I’Il actually take you through the data 
structure and tell you more of what all of this actually means. I’1l show you questions, Ill 
show you sub-topics, etc. So, how do we go about creating these debate maps? Which, again, 


I will show you what they actually look like towards the end. 


We have this process that we’ve developed. Essentially: 


* it starts with archiving and collecting mass content, 
* transcribing it and standardising it to text, 

* extracting arguments, claims and then evidence, 

* categorising those, 

* clustering those into hierarchical categories, 

* inputting that structured content into a database, 


¢ and then tinkering with visualisation so that we can compress as much knowledge as 
possible in visualisations that can convey the complexity, without being way too 


overwhelming. 


Something that we’re really interested in is: How do we create knowledge compression 
where people can see as much knowledge as they want to see and they have the flexibility to 
work in various dimensions to unpack what they want to explore as they want to explore it? 
So instead of an author, who’s writing a book or a paper, taking a reader through a 
specific narration, instead it’s about: How can we visualise all the possible narrations that a 
reader can go through and they can unpack in the direction that they want to? And I think 
we’ve had a recent breakthrough in how we’re going to go about doing that in a really simple 
way, so I’m really excited about that. I’ll show you a tiny little sneak peek of it. But most of it 


is on wraps until we launch the project at the end this month if all goes according to plan. 


So that’s the basic process, but let’s get specific because that kind of matters. So when 
we talk about archiving what do we mean? Well, first we built a bunch of custom search 
engines, to essentially, make sure that we’re pulling from all across the political spectrum and 
across different forms of media. We also have curated feeds that we keep an eye on. We are 
also aware that it’s really important to break the digital divide. There are just some things that 


people are not going to be willing to write a medium post about or post on Twitter about ideas 
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that they have, that they do want to express. So, facilitating conversations and recording those 
interviews with permission, in order to gather that audio data is very important. We’ve also 
acquired access to various searchable databases, GDELT, and Internet Archive, which have 


just been absolutely wonderful to us. 


One thing I do want to emphasise is that The Society Library takes very seriously 
our research methods. We have over 22 methods that we developed to try and counteract 
our own research biases. On our own echo chambers, we have a list of policies, also, on our 


website that talk about all the wicked problems of knowledge management that we’re seeing. 


¢ When it comes to inputting content in the database, are we going to platform a dog whistle 
that we’re aware is a dog whistle? 
¢ Are there going to be policies about trying to referee if one group is calling another 


group’s language a ‘dog whistle’ when, perhaps, it isn’t? 


There’s all these wicked issues that we care very much about in addition to the methods that 
we’ve already deployed in order to overcome our own biases. And we’ve got a virtues and 
value page related to how we see ourselves in relation to knowledge. When it comes to us 
digging and scaring around, and archiving all this content, how do we feel about 
misinforming and disinforming content and things like that it’s, we like to approach our work 
with total intellectual humility, and those virtues and values are listed on our website. So we 
talk about that often in the culture of the Library, we often hire librarians who have their own 
code of ethics, which we also appreciate very much. The librarians in general, in the United 
States, take a very anti-censorship stance and instead argue that, if there’s the right context, 
any information can be interacted with to enable enlightenment. It’s not that some 
information should be hidden away, but what is the right context for knowledge to be 
experienced so that it informs and enlightens, rather than corrupts and persuades. So, we have 
developed our own code of ethics. We are also members of the ALA, so we adhered to those 
rules as well. I just love to say that. And then the process also begins by this, kind of, flyover, 
we call it, where we quickly find a whole massive topic, this can be done manually or 
computationally, where we just collect a bunch of topics across various media types, and we 
do that by specifically targeting and grabbing sample collection across different media types 
that contain certain keywords that are related to certain topics. If we’re mapping something 
related to nuclear energy and tritium leaking into the environment, which is a radiological 
hazard is one of those topics, our archivists and librarians are going to go through and find 
where does that keyword happen in podcasts, books, definitions, and government documents, 


etc, not only through time but also from news articles that are across the political spectrum. 
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Once we have that diverse set, that’s when we do the next step, and let me just quickly say 
where we collect content from, scholarly, articles, research papers online, news, websites, 
blogs, social media, Twitter, recently Tik-Tok, Facebook and Reddit, we pull from 
documentaries, videos, and television, topics, specific community forums, and groups 
conference, videos and summaries, government publications and websites, existing FAQs and 
online resources. And then, we also conduct interviews with industry leaders, thought leaders, 


and experts. 


And pro-tip, so many government agencies in the United States actually have in- 
house librarians and they are so helpful. We just send them our research questions like, 
“Hey, can you look through your entire agency’s library and help us find the relevant 
documentation?” And oftentimes they’re more than willing to help. So that’s just really 
lovely. 

Anyway, once we grab the sample sets of data, we deconstruct all of that into text, we 
transcribe it, we translate it, we parse it, sometimes we even hire people to actually type up 
descriptions, text-based descriptions of videos, and graphic imagery, so we have that text, so 
it’s searchable. And then, once we have that content, what we do is, we start deconstructing 
the arguments, claims, and evidence. So we’ ve got a training program for that. We have our 
own standards for what we mean by claim. Drive claim, implied claim, implicit claim, 
argument, etc. And I’m going to show you what deconstruction looks like just because it’s 
good to know what we’re talking about. So this is a transcript from a Sean Hannity? clip. It’s 
an old example, but it’s one of my favourite examples, because one of our favourite people at 
the Internet Archive asked us, specifically, to deconstruct this, because they couldn’t believe 
that Sean Hannity would make all these claims. I think this is only about like 17 minutes in 
length, yeah, it’s about 16 minutes and 20 seconds and all the claims that we were able to 
extract, I think, let’s see here, 100, oh, the implied ones are hidden away, so, in terms of 
directly derivable claims, Sean Hannity made 179 claims in his 17-minute snippet, and there 
was way more implied, I accidentally pulled up the wrong example, so sorry about that. I’m 
not an excel spreadsheet wiz, so I don’t know how to unlock the hidden implied claims. But 
anyway. From the exact transcript, which is right here, we actually pull out the arguments and 
claims directly, so oftentimes, you can see that there’s one line right here that can actually 
pull various claims from that. And that’s because language is complex. It’s dense. And we 
really want to extricate all of those tiny little fundamental units of reason, because we 
actually want to fact check it, qualify it, debunk it, ‘steel man’ it, devil’s advocacy, at all 
these things. 

Another example I’d like to show, this is the Green New Deal. The Green New Deal 


is a famous legislation in the United States. When we’re training our students, because we 
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have various educational internships, we’ve worked with 32 universities in the United States. 
One of the training projects the students do is, they deconstruct the Green New Deal to, kind 
of, get a handle on what the Society Library standard for clean is. So I think there are 438 
claims in the Green New Deal. It’s a very short piece of legislation. If you just copy-paste it 
to a Google Doc, it’s like, 13 pages, I think, in a relatively big font. So it’s a relatively short 
documentation, yet, there’s 438 claims. And there’s very little evidence that’s usually 
provided in the legislation. It usually qualifies itself within like the... It actually does have an 
interesting structure. It’s a very complex argumentation. It has a premise where it says, 
“Congress fines given this document”. Which they reference the IPCC report if I’m not 
mistaken. These are our findings and our conclusion, which is the recommendations by 
congress to create a specific policy program or whatever. So, yeah. We deconstruct pretty far, 


I would say. 


And then, going back to the presentation, what happens when we have all of these 
claims? That’s when we start categorising them. So these claims are going to have keywords, 
those keywords are going to be semantically related to other keywords. There’s ways in 
which, I hope in the future, we’re going to be better at computationally clustering these things 
together. I’m really interested not only in taking the data that we’ve created, using it as 
training data for claim mining, but I’d also like to start seeing if we can generate syllogisms 
just by the relationship between the keywords in text snippets. So that, potentially, with 
enough training, maybe our analysts would have more of a fact-checking role than 


constructing arguments from the base claims role. 


But technically what we do is, we categorise cluster claims based on the relatedness to 
specific topics. So they may be, for example, on the topic of nuclear energy, it could have to 
do with grid reliability or stability, the tritium leakage, other radiological issues. We just 
cluster those into categories, and then from those categories, we’re able to derive different 
positions and more complex argumentative structures. I’m going to show you what some of 


that looks like in the debate map as we go on. 


And I will say also that we’ve been very fortunate to have a very large tech company, 
who we’re not allowed to name, and a lovely university who we’re not allowed to name, who 
have made fantastic argument mining technologies, and they’ve given it to us to use. But 
we’re a small non-profit, so they’re like, “Yeah, don’t tell anyone we’ve done this”. So we 
also have interesting argument mining tools and we’re hoping that the training data we’re 
creating can make things even better. And this is just an example, again. 

In this one natural language text snippet, we can pull all these claims. 


A derivable claim means that we can, essentially, use the same language in the text 


snippet and just cut out some things and reconstitute it, in order to create claim. 
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When an implied claim is that you have to have some, sort of, insight into the meaning 
of the claim itself, which requires human intelligence in order to suggest that this claim 
would have to be a part of the argument, or one of the premises of the argument, in order for 


the claim to be proven to a certain extent, or made sound or valid. 


So our analysts are also trying to put in implied claims but also mark them, so they 
don’t get confused with things that have been derived from evidence or from sources. That’s 


just an example of that. 


And then, understanding our hierarchy is really important too. What we found is that if 
we just randomly choose a question, or randomly choose some dimension of a debate, what 
happens is, as we’re hand mapping the logical argumentation from that one point in the 
debate, we start to quickly get into a spaghettification problem. So we start having 
arguments that are somewhat relevant, it just, kind of, spiders out, and curls in on itself. It’s 
very messy. From what I’m told from people who’ve worked in AI for a long time, it’s called 
a good old-fashioned AI problem. But what we’ ve discovered is, if we just do this kind of 
hierarchical clustering, over time, essentially, what we can do is have this descriptive 
emergent ontology that occurs. And what’s interesting is that the questions that are derived 
from finding the references and evidence, extracting arguments in the claims, organising 
those into those categories, and then, identifying those positions. In finding those questions, 
the questions, in turn, shape the relevance of what can be modelled in response to the 
question. So if we’re interested in having the most steel man formal deliberation possible, it’s 
the responsibility of our analysts to make sure that the positions are actually answering the 
question. So finding the questions that give shape to the relevance of the argumentation, 
which has really helped us to avoid that whole spaghettification problem. I don’t know if it 
solves that good old-fashioned AI problem in general, but it’s just something that has really 
helped us. We call it descriptive emergent structuring. And we’ve used it on all of our debates 
since we learned the first few tests that, just picking a random part of a debate isn’t going to 


work for us. 


And then, we go ahead and map this content. We have a debate mapping tool. Every 


single week we ship new features and make it even better. 


Some of the things that we can put into this debate mapping tool includes: 


* Question nodes 


* Category nodes 
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¢ Argument nodes 
¢ We can create multi-premise arguments, unbelievably rigorous logical proofs. 


¢ There are claim nodes. 


We can also, within every single node, have: 


¢ DefinitionsVariant phrases 

* Videos 

* Images 

¢ Media 

¢ Equations 

* References 

* Quotes 

¢ There’s ways in order for people to participate in calculating impact and veracity scores. 


¢ And then, every single claim and node can have a pro, con, truth, relevance argumentation 


associated with that, as well. 


So we’re able to have a pretty complex argumentation that’s mapped out using this specific 
tool. And then, of course, we want to model it. And like I said, of the knowledge products that 
the Society Library produces, creating these maps is just one thing. We’re really interested 
in creating a much more accessible visual libraries apps. I would love to put it in VR, 
how fun would it be to stretch and open claims, to unpack them? It would be lovely. I 
would love to take some of our data and maybe work with someone who would have an 
interest in visualising these things. I think it could be lovely. 

But the other things that we do, as I mentioned, we create decision-making models, so 
we’re moving into the smart city space. We’re trying to pitch it like, smart cities could be 
smarter if they had ways to augment their intelligence by externalising the decision-making 
process. Some of the work that we’ve already done, I mentioned earlier that, something that 
we’ ve discovered in our work is when you really map a space, you find out how dense and 
complex it is. 

So a city council wrote to us, and they asked us to help map a debate that they were 
having locally. And they thought it was a binary issue, like yes or no. And we found there 
were over 25 different dimensions of the decision that they were facing. And there were 


anywhere from two to five arguments with or without evidence in each one of those 
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dimensions that they would consider. So we created a micro voting protocol that essentially 
allowed them to zero in on one dimension at a time. And the feedback that we got from that 
project was just so wonderful. People who felt like they were being marginalised, felt that 
they were heard. People who felt like they were on the fence, we were finally able to see that 


they were actually really certain of a specific position, they just need to externalise it. 


So we’re moving in that space, and we’re pitching to various city councils to, 
essentially, not only do research work for them, like the Library of Congress does for 
congress, but also create these decision making models because that can increase 
transparency, accountability, and decision making, and it helps overcome a little bit of 
cognitive bias. Because if we’re making decisions in our heads, who knows what kind of 
black-box calculations and waiting is actually going on. 

But if we’re forced to externalise it, and work with things one-on-one, and actually 
identify what dimensions we’re in agreement with, it just, I think, helps improve the process 
in some way at least a little bit. So we’re hoping that we can just improve decision-making in 
general. 

And then, we’ve also been hired to create legislation. So using our method of 
deconstructing content down to the claim level, we produced federal level legislation. We 
essentially took hundreds of pages of congressional recommendations, broke them down to 
the claim level, and then, we compared those with legislation that was passed, failed, and 
pending at the state and federal level in order to, essentially, say, what, in the congressional 
recommendations, on this specific issue, is missing from the existing legal code at the federal 
level, and then, borrowing language from where it’s been attempted in the past, and produce a 
bill, just essentially, by claim matching, and filling in the blanks. So we were willing to work 
on something like this because it’s a very non-partisan issue. It had to do with the 
infrastructure bill. It had to do with the integrity of the electrical grid in the United States and 
recommendations to harden or make it more resilient. So it’s a totally non-partisan issue, we 
were happy to do it. And we also got amazing feedback from that work. We were able to, 
literally, deconstruct hundreds of pieces of legislation, and hundreds of pages of 
congressional recommendations in under three weeks, and deliver this proposal. So, most 
likely, we’ ll continue doing non-partisan legislative work. 

And then, also, we think that some of the data sets that we’re creating may actually be 
very useful training data, not just for us, but for other people as well. We’re thinking about 
that, also, being a potential revenue stream as a non-profit. 

Oh, one more thing. I mentioned before that this debate mapping software has a lot of 
different features. We can pack videos, images, quotes, equations, references, and all these 


different things in a single node. And so, when we made a submission to the Future of Text 
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book, it was about our concept of web-based conceptual portmanteau. I’m going to show you 
a little bit of what that looked like when we were initially mocking it up, and then by the end 
of this presentation, I’m going to show you where we’re at currently because I think it’s a 
huge improvement. 

So our first foray into web-based conceptual portmanteau is, we were, essentially 
saying, portmanteau, when you combine a couple of words together, and it’s like this more 
complex meaningful word. And we’re like, “Okay, well, we’re trying to create portmanteau 
in terms of media assets. We want to combine references with images, videos, text and 
different variant phrases of the text, and all these different things”. So we mocked up this new 
node structure. And we were talking about different ways in which people could unpack and 
repack all of that knowledge. I’m going to show you our debate map and take you through a 
little bit, and then, I'll show you the new version of web-based conceptual portmanteau, 
which is still being a little like tinkered with currently. I’m happy to show you a little bit of 
what we’re doing. So here’s an example of one of the debates that we’re going to be pushing 
out. This is the data staging area, so you’re very much seeing the behind-the-scenes. These 
are the tools that our data brains and librarians use to input structured knowledge. 

I’m going to show you how complex it gets. So we were asked to map the 
deliberation about the Diablo Canyon Nuclear Power Plant in the United States. It’s the 
last one in California that’s in operation and there’s a great big debate about it. I think even 
Elon Musk has alluded to it in tweets. So it’s a high-profile issue for the state of California, 
and we were very lucky to be asked to start mapping it before it really blew up. What we’ve 
found so far is that there are generally like two fundamental questions that the community has 


which is: 


¢ What should happen to the Diablo Canyon Nuclear Power Plant? 


¢ And why is it being decommissioned? 


I put little trophies as notes to myself so I don’t get lost. And so far what we found is there’s 
about seven different positions that the community is taking on these issues. And by 
community we mean academics from MIT, we mean the governor, we mean activists, we 
mean members of the local community. Essentially we had an interest in finding all the 
different stakeholders, checking out the media that they were producing, and then, extracting 
that media. 

This data set, I believe is drawing knowledge from 880 different media artefacts. I have 
a list here. In this database, there are references to 52 knowledge from 51 scholarly articles, 


eight TV segments, 112 reports, five books and textbooks, 367 news and websites, 194 social 
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media posts, 66 videos, 24 podcasts, 53 government international documents, together for 
880. 


And so far what we found is, there’s seven general conditions. One of those is that it 
should just be left to be decommissioned as scheduled. And then, we see that this breaks up 
into a variety of different categories. So there’s economic issues, environmental issues, safety 
and well-being issues, ethical issues, all in support of why it should be decommissioned to 
schedule. So, as we unpack this, we can actually start seeing some of the reasons that people 


pose for why it should be commissioned. 


And you may notice that there’s these little brackets right here. Those brackets are important 
because, in order for us to translate data from this data set to the visualisations that we’ve 
created, we’re actually substituting these lines, which are indicators that the relationship 
between nodes with this language. So we’re doing this, even though we’re just creating this 
knowledge graph structure. And these lines, their colours, and orientation are indicators of 
their relationship. We actually will substitute with that language, so it’s completely readable 
(indistinct) visual. 

Some of these claims in support of it being decommissioned that are related to 
economic issues include: that closing the plan won’t harm the local economy as much as 
previously thought, and that market forces have made the Diablo Canyon Nuclear Power 
Plant redundant and uncompetitive, so we can unpack this even more. And again, in brackets 
means that we’re essentially identifying the relationship between these tech snippets, other 
tech snippets, so as we move on from this one which won’t hurt the local economy as much 
as we thought, the economic impact of the local economy would be smaller than previous 
estimates in part due to economic resources that we made available, and over time, the region 
will overcome economically and experience positive growth. Now I’m just going to take a 
moment to start unpacking a little bit of what’s available in this node. So what you’re seeing 
is this text and these references here. What we also actually have is a bunch of different 
phrases, a bunch of different ways of expressing the exact same point in a similar linguistic 
register. So we’ve identified that as standard. We also want to make all of these nodes really 
accessible to people with less subject matter familiarity and can handle less cognitive 
complexity. So this simple version uses much more simple vocabulary. It says, even though it 
seems like there will be a lot of bad economic consequences when it closes down, some 
experts think it won’t be that bad. So it’s a very simple way of expressing the same thing. 
And we have some more technical versions of this claim that actually refer to the economic 
assessment. Why that assessment was commissioned? What type of specific input-output 


modelling tool was used to derive those conclusions? It’s a much more technical way of 
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expressing that same claim. And then, in support of this particular claim, we have a multi- 
press right here which actually breaks, this is the summary of the multi premise which we can 
unpack here that actually breaks down the logical argumentation that led to this particular 
conclusion. So every single stage of, not the experimentation but the conclusion, so the first 
step of the decommissioning process will result in this kind of economic growth, and we’ll 
have economic losses, then we’ll have economic benefits. Overall it will conclude that. And 
this can unpack even further because each one of these premises in this argument can also 
have pro/con argumentation that can argue whether it’s true, support it, or its relevance. We 
have an example of a relevance argument here, which calls out that the economic impact 
assessment didn’t look at the economic impact after the decommissioning of the plant was 
complete. So when it talked about positive economic growth, it was only so far out of a 
projection and they didn’t include certain things in their model. And, of course, this comes 
from news articles. So it’s about collaboratively finding where the conversation is happening, 
distributed across different media artefacts, and then bringing that argumentation actually 
close together in one spot for people to explore. I'll show another little example and there’s 
tons of these. Based on market forces demand for CCP nuclear energy is expected to decline. 
We have another multi-press here which essentially breaks down all of the different economic 
trends that are happening, which have been collaboratively expressed. So it’s not just that 
energy efficiency policies are going to reduce overall electricity consumption, there’s also 
increased solar then CCAs also don’t prefer to buy nuclear energy. And as you can see, it can 
get pretty dense pretty quickly because all of these are unique arguments that support these 
more vague generalisable premises. So the more that we go upstream and go left, the more 
vague the statements are, and the more interested someone is in to, actually, digging into the 
argumentation and evidence that’s available to support these vaguer, high level, commonly 


expressed sentiments, people can really dive in deep and explore. 


But of course, this begs the question: Why on earth would anyone do this or want to 
find knowledge organised this way? 
And even though when we publish this collection, we’re going to have a viewable version of 
this map accessible if people love to explore this map. Hopefully, we’ll deal with the 
functionality a little bit, we will centre things and make it a little bit better. But we know that 
this isn’t really going to cut it, in terms of making this accessible, legible, etc. So what I’m 
really excited about, going back to the web-based conceptual portmanteau, we’ve been 
wondering, how can we take all of this knowledge and really compress it down? So I’m 
going to show you a sneak peek, I wasn’t originally going to show up because this is 
recorded and I didn’t want this to end up on YouTube yet, so I’m just going to show you one 


image and just give you a little taste of it. 
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This is what we call Society Library papers. This is just the design mock-up, we’re 
building it right now. But we’ve figured out a lot of different features that I’m really excited 
about, and we’re hopefully going to have more time to test it and make sure that it’s actually 
as visible as we think it is. Actually, a lot of knowledge can be compressed in a paper, and 
what we’re doing is actually allowing the paper itself to unpack in different dimensions. And 
by clicking on different snippets, each one of these are arguments. It can open up different 
screens where people can view the variant phrases and things like that. So we’re going to 
drastically simplify and compress a lot of this knowledge while maintaining all of the 
complexity. But making it more of like a bulleted outline that opens up and closes back up as 
people want to dive in deep to each one of these sections. And we want to have ways in 
which people can track how much of it they’ve seen, etc. There are other ways, in which, 
we’re thinking about visualising content too. But that still needs to be flashed out more 
before I’m excited to share it. Although there are things I really think, just like we are 
figuring out all the ways in which we can have all the complexity and features of this debate 
map in a piece of paper, I think we’ll be able to do the same with video, and if some magical 
VR magician comes by and wants to take this data and start changing visualisations as well, I 
think experimenting with it in VR, is going to be very important for us as well. So I think 
those are all my tabs that I had an interest in showing you all. And 45 minutes in, wow, I 
think it’s a good time to pause and ask: 

You all have been thinking about these issues longer than we have, what should we be 
learning? What have we missed? Who should we be learning from? Would love your 


feedback. And thank you all for your attention and time, too. 


Dialogue 


https://youtu.be/PucSvzwp81Q?t=2030 


[Frode Hegland]: Yeah, thank you. That was intense, and our wonderful human transcriber 
is going to work overtime on your presentation. Danillo, he is very good, so it’ll be fine. ’'m 
going to start with the worst question just to get that out of the way and that is: You’re 
American, we’re British. You had Trump, we have Boris Johnson. It seems a lot of that 
politics is just personality-based, “Oh, I like him”. Or some kind of statement like that. 
Where would you fit that in here? Or do you consider that, for this, out of scope? 


[Jamie Joyce]: To a certain extent, I think there are certain things that are relevant, and some 
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of it is out of scope. So, one thing I just want to acknowledge is that I think we’re in the 
middle of an epistemic movement. So there are a lot of people who are working on different 
dimensions of how can we have an epistemic revolution? I like to call it the e-Lightenment. 


So how can we use new technology to transform our relationship with information? 


[Frode Hegland]: Hang on, e-Lightenment? Has anyone else on the call heard that 
expression before? That’s pretty cool. Let’s just underline that. The e-Lightenment. Okay, 
that’s wonderful. 

[Jamie Joyce]: That was the name of my TED Talky, yes, I called the e-Lightenment. 
[Frode Hegland]: Oh, no. You caught us not having seen your TED Talk, now we have to 
watch it. Okay, fair enough. 

[Jamie Joyce]: No, no. It’s old. It’s not amazing. I really want to redo it. If I redo it, ’m 
going to call it something like Big Data Democracy, and talk about the complexity, volume, 
and density of our social deliberations, and how we need new tools to really experience our 
big data society in the way in which it actually exists in reality. But anyway, yeah, so I think 
some things are relevant, some things are out of scope. There are a lot of people who are 
working on different dimensions of these issues. And one of the contributions that The 
Society Library is making outside of knowledge projects is also education. We’ve been 
working at the university level, we’re trying to bring it down to high school, and then we’ve 
been collaborating with some people ideating how we can start an epistemic appreciation, 
learning about cognitive biases, logical fallacies, and the various ways in which you can be 
disinformed on the internet into a younger and younger children’s programming to develop 
literacy, and a standard for what we should appreciate, in terms of, high-quality work versus 
not high-quality work. And obviously, there’s people who are working on making social 
media less effective and less addictive. There’s lots of different people who are working on 
this. And you’re right, I think a lot of politics is about personality. So what I think about often 
is, “Okay, well. How do we, essentially, make smart really sexy in the United States?”. So 
once we have these knowledge products, you have to create the demand for people to want to 


use them. So, what kind of people need to associate with these knowledge products? 


[IN CHAT) From Peter Wasilko]: Do you flag logical fallacies in the presented text? 
[Jamie Joyce]: Yes, we do. So I didn’t mention it. Thank you for your question, Peter. We 
have a tagging feature and we use our tags and it actually appears on the paper. So when 
someone unpacks a node in the paper and we have a tag on it, it appears as a handwritten note 
off to the side in the marginalia that just lets people know like, Hey, this is an opinion. This 
needs to be checked. This is cherry pick data, etc. Sorry, I saw your note and just wanted to 


answer that really quickly. So how do we create the personalities? You know, fictional or not? 
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These could be in kids’ shows, for example. How do we create the personality and personas 
that are sexy and attractive that are pointing people towards the cultural values of 
appreciating more rigorous research higher standard for argumentation and these sorts of 
things? And some of our donors and supporters have been thinking about this also. And 
thinking about supporting subsequent and related projects to help drive up demand for people 
appreciating this. And something that we see in the Trump era in the United States is that 
there’s been a huge decrease in trust. I think this was happening well before Trump, I think 
Trump was a consequence of this happening. But I think he also helped make it a little bit 
worse. There’s been mass amounts of distrust in existing knowledge institutions. Like news 
media, universities, government agencies, these sorts of things. Some sections of the 
population are not as trusting to get their information from those institutions. However, I 
think libraries, very interestingly, have maintained their level of trust in American society. So 
we do recognise that there is an element of branding and storytelling to be attractive to the 
community. It’s probably going to be very long-form relationship development. And that’s 
one of the reasons why the Society Library takes its culture so seriously. We take our virtues 
and values so seriously because we are going to be an institution that isn’t going to get 
thrown away immediately. That means we have to always have the out-facing 


communication, the branding, look, and the integrity to earn and maintain that trust. 


[Frode Hegland]: On that issue, on the fake news. This [holds up book] (Snyder, 2018) is a 
phenomenal guide to fake news, as opposed to propaganda. It basically makes this simple 
obvious statement that, when Russia first invaded Ukraine, the point of fake news was not 
wrong news. It was simply wrong and true mixed, so no one would believe the media. And, 
of course, clever people like to think, “Oh I don’t trust the BBC”. And, you know, the 
situation we’re in today, which is pretty awful. And then I have a very specific semi-technical 
question, this goes back to having conversations with Marc-Antoine, of course, and that 

is, the last thing you showed, that normal document where you can click and things open, that 
is, of course, phenomenal, and it is something that, we in this community, we really like the 
idea of being able to get a summary and then digging into it. So my question to you is: In 
what way is it open and interconnected? Can I use it in my academic document? Can 
Fabien use it ina VR environment? And can Marc-Antoine, I guess you can, extract it into 
his knowledge graph? How does this data move around? 

[Jamie Joyce]: Well, good question. So all the data coming from the debate map can be 
referenced and extricated elsewhere. The paper document is so brand spanking new, we 
haven’t even thought about integrating it with other platforms. So we’re still wrapping it up 


as we speak. When it’s finished though, I would love to start inquiring into, how it can be, not 
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only maybe productised so other people could use it, essentially, it would require a different 
interface to input data. Most likely because I can’t imagine people are going to quickly get up 
to speed with our really complex debate map. So creating a user-facing product input form 
into a structure that will probably be more helpful to others. So I don’t have an answer to 
your question yet. I would love for it to be productised and for it to be ported elsewhere. But 
the debate map does, that data can be extracted and referenced and all of that through an API. 
[Frode Hegland]: Marc-Antoine, do you have anything to add to that? 

[Marc-Antoine Parent]: We certainly both believe in the value of making these new ways of 
expressing information, both in continuous text, in graphs, and making them interrelated. 
How interrelated, there’s many models. And I think we’re both, separately and together, 
exploring ways to do these interrelations. Certainly, the ability to tag concepts or arguments 
in text, I doubt very much that it won’t be connected to a graph realisation. In that way, if you 
have an export from the graph, the question is: Can you identify these things in the text 
document, right? And then we can speak about offline annotation. We can speak about 
edition. We can speak about... Somebody mentioned stretch text in the notes, yes, I believe in 
that. I believe in side-by-side views, personally. These are having the graph with the text 
coordinated, that’s something I’m pursuing. As I said, I’m not part of that team. I don’t know 
how Jamie’s doing that part. I am helping her more with the extraction a bit, so, yeah. 

[Jamie Joyce]: Very much so. Well, you can join the team, Marc-Antoine. I’d love to get 
your thoughts. We’’ve just been so swamped in the world of design, I didn’t even think about 


tapping on your shoulder. But I always love working with you. 


[Frode Hegland]: Yes, you two, yes. So, okay. I'll do something controversial, then, and 
show you something. Just briefly. most of the people here know this all too well. This is the 
most poverty-stricken thing I could possibly show you. But it’s about an approach, not a 
specific thing. I’ll do really briefly. At the beginning of documents in a book, you normally 
have a bit of metadata. PDFs, of course, normally never have anything. So this approach 
that we call ‘Visual-Meta’, is to take metadata on the last page, right? This obviously wasn’t 
made for you, so I’ll just show you a few brief things and mention the relevance. It is 
formatted to look like BibTeX, and that just means it looks this is this, this is this, really, 
really simple, right? So this example here happens to be for the ACM Hypertext Conference 
last yeart and this year. But the idea is that, all we do, when we export to PDF, is to write at 
the back of the document what the metadata is. And that includes, first of all, who wrote it, 
because very often, an academic article, when you download it, you don’t even know the date 
that was published, because it’s from a specific bit of the journal. It also includes structural 


metadata, i.e. headings. They can also include who wrote the headings and what levels. And 


330 


then they can include references. So all of this is in the metadata that we then take into VR, or 
wherever, and use it. So one thing you might consider, and this is something we’d love to 
work with you, but if you do it entirely by yourself, that’s fine. All this stuff you have, when 
you do that top-level presentation, just stick it in the appendix. As long as you explain in the 
beginning what it is, in normal human language, let’s say, in 500 years when someone comes 
across the PDF and everything else is dead, they can reconstruct it. 

[Jamie Joyce]: Yep. I checked out some of it, I think one of your explainer videos on it. I just 
got to say, I absolutely love it. I love it. 

[Frode Hegland]: I’m glad you do. And thank you. I mean, every couple of months there’s a 
circle of arguments of, “Oh, we shouldn’t use PDF”. We don’t just use PDF. It depends where 
you’re doing stuff. When we go into VR, we use different formats. But at the end of the day, 
you’ve got to archive something. And that’s why it’s used by billions of documents. 
Somebody will keep it going. So when it comes to the finish bit, yeah, you know that whole 
workflow. I’m just glad we had that little back and forth. Any other questions? And by the 
way, Daveed and Karl, it looks like you’re wearing the same hat. It’s so funny. Because of the 


green background. 


[Daveed Benjamin]: Oh, that’s funny. Yes. Nice. I have a question, Jamie. Where are you 
headed with the visualization on screens? 

[Jamie Joyce]: We’re working on creating this multi-dimensional explorable and interactive 
piece of paper. And then, I think we’re going to move on to recreating the newspaper and 
recreating TV as well. Because again, all of those nodes can be the ones that have video 
content because we clip it where the expression of the claim is associated in video with the 
node itself. So, as especially we get more and more sophisticated with automating some of 
our processes, and making sure each one of the nodes are actually multimedia, I think the 
same way of compressing and compounding into a dense layered interactive set can be 
translated across medias. So I’m really excited about that. But again, I cannot state enough 
how interested we would be in creating a VR library, because I think that would be so 
exciting. Or a VR debate. I think that’s really important. So we’re just tinkering right now. 
And we’re just finishing up the last really complex argumentation structure and creating 
corresponding paper features and visualisations for that. And we’re looking to push it out by 
the end of this month. And then we’ll test and get feedback and see how useful it is and all of 
that. 


[Daveed Benjamin]: That’s super cool. I look forward to seeing that. 


[Frode Hegland]: Yeah, that’s really wonderful. I see Fabien has his hand up, which is great 
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because I was about to call on him. I was just going to say two things in context for Fabien. 
Number one, we’re doing some basic murals in VR now. And even just a flat mural is really 
powerful. And we’re looking at all kinds of interactions. But also, we do meet every Monday 
and Friday. All of you should feel free to dip in and out as you have time, because right now 
we’re at the stage where we’re learning how to do folding, or this, and that. So we’re at the 


detail level which could be really quite exciting. Fabien, please. 


[Fabien Benetou]: Thank you. And thanks for the presentation. What I wanted to say, I have 
a presentation due for this group named “What if a librarian could move the walls?”. I 
think it should pique your interest. But I’ll give a little spoiler for this presentation, which is 
to say, in my opinion, even if, for example, your information or your data structure is very 
well organised, it might not be the most interesting for participants, because that might 
become a little bit boring. If it’s too structured, let’s say, if you go to a hospital or a large 
public building, if every floor is a copy-paste of the other, we get lost, basically. So being 
structured is extremely powerful. And we can process and we can do quite a bit with it. But I 
don’t think it’s sufficient. It’s not a criticism in any form or way. I’m just saying, today if you 
give me the data set, I can definitely make an infinite corridor, a very long corridor, with all 
the information. But, yes. I think it would be fun to do, but I think it’s not sufficient. I think 
you will experience it, have a form of way to be through it, but one of the, at least my 
motivation for VR porting of text, documents, or information, is how smart our body is, and 
how we can remember when we’ve been to? Like I was mentioning a bakery in Berkeley, 
because I haven’t been for a while, but I did go and I remember how to get there, and how to 
go in the bathroom of a friend. This mind-blowing stuff that any of us, every one of us can 
do. But because we have some richness of the environmental diversity, So it’s a bit of a word 
of warning to say, porting it to VR it’s definitely feasible today. It’s not a problem. It would 
definitely be valuable and interesting. But it would probably be quite interesting or more 
valuable to consider what 3D assets you do have. Is there actually a structure behind, let’s 
say, an argument map? Can you actually visualise it, not to visualise it, but in a way that you 
mentioned knowledge is compression that you can synthesise in a way that is meaningful. 
Not just to specialise it in order to specialise it, but to specialise it because that mapping or 
that visualisation makes sense. So I think that’s a little bit of a challenge there. And again, I 
say this candidly or naively, I don’t have sadly an obvious or immediate answer to this. But in 
my opinion, that’s where the challenge would be. 

[Jamie Joyce]: Yeah, I absolutely completely agree. And when you mentioned how 
intelligent our bodies are, what immediately came to mind is how, when we were looking at 
different visualisations for the data on, essentially, a 2D screen, where we can’t interact with 


it in a three-dimensional space, so to speak, I was trying to find a lot of inspiration from 
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video games. So, I was looking at a lot of video games, of how they compress knowledge and 
organise it. They have all their accessories, and this is how they upgrade their armour, and 
blah, blah, blah. So I was looking at tons of those. And what I was really interested in and 
inspired by, were these things called star charts. So there are ways in which people can 
develop their character, where they move through a three-dimensional space of lighting up 
nodes and it essentially shows that they’re headed in a specific direction, it shows them 
enough of what’s ahead of them, so they know to move in a direction, Final Fantasy does 
this, for example. And they also know there’s a whole other section over here of undeveloped 
traits because their character is moving in a specific direction. And so when I was thinking 
about our argument maps, if we were to take those trees, and essentially lay that out in 
physical space, is there some kind of metaphorical thing that we can pull from real life that 
would map onto people’s brains really easily so that they could use their geospatial 
intelligence to, not only remember the content but be interacting with a little bit more? And I 
couldn’t help but think like, essentially roads. So all of these different signs that would 
indicate if you want to go to economic here, blah, blah, here, blah, blah, blah, here. And they 
could be taking a walk through the debate. And it could be an enjoyable experience because 
there could be lots of delightful things along the way that they could be seeing. As they’re 
walking in the direction of the economic arguments there’s another signpost with all the 
different signs pointing in different directions, they take this one, etc. There’s the map up on 
the corner showing them what territory they’ve explored. This is something that exists in 
video games. And I mean, given the structure of our data in debate map, it seems as though so 
long is similar to how we are designing all of the assets in the paper to unlock, unfold, split 
up, and blah, blah, blah. If those similar assets could be rendered in a 3D space, you could 
just map out entire territories of physical traversable space. So I don’t know, given that 
you’re an expert, if you think an idea like that could be worthwhile. But I completely agree 
with you that just visualising it as books or something on a shelf isn’t going to do it, because 
we’re a different kind of society now. And so, part of the reason why we conceptualise this 
idea of a web-based conceptual portmanteau is that we know that we have to develop new 
media to express knowledge. We can’t just directly digitise books, or pages, or essays, or 
newspapers anymore. It’s multi-dimensional. 

[Frode Hegland]: But there is a really interesting ‘however’ which we experienced recently. 
Bob Horn gave us a mural that we put in VR. And if you pull it towards you and push it, no 
problem. But if you walk it’s so easy to get queasy. Motion sickness, for so many of us, can 
happen. So the idea of walking down a road it’s great for some, but it can easily just not go. 
But also, before I hand it over to Mark here, Brandel forced me to buy this book last time by 


holding it up, that’s how we force each other here. And the introduction is not very good. It 
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really threw me off, because she’s a journalist. However, when you get to the chapters, the 
embodied thinking and so on is absolutely phenomenal. I think you would greatly appreciate 
it. But, yeah. The reason I highlighted that point is me having been in VR only three months 
now. Properly on and off. I had so many preconceptions that are just getting slaughtered. So 
to work with Fabien, with such rich deep experience, and also with Brandel, who’s working 
on these things to really learn to see differently, we have to re-evaluate this. For a while, we 
call what we’re working on, Metaverse. But just looking at the proper definition, that’s all 
about the social space. What we’re doing in this group is not so much social and definitely 
not gaming as such. It is about working in virtual environments. And it seems hardly 
anybody’s focused on that. So for you to come into this dialogue with actual data, actual use 
case, actual needs, that’s a really wonderful question. So, I want to thank you very much. 
Jamie Joyce: Yeah, and thank you for that advice. And thank you, Fabien, too. So, are there 
meetings that maybe I could sit on so I could benefit from this sage insight and experience of 


translating things to VR? 


[Fabien Benetou]: Sorry to interrupt, but to be really direct, did you put up your headset 
for the last couple of years? 

[Jamie Joyce]: It’s probably been about a year since I’ve put one on my face. 

[Fabien Benetou]: Okay, but you did. So that’s fine. Because, I think, honestly, with all the 
due respect to everybody around here. Putting their heads at once recently is more valuable 
than any or all of our meetings. And then coming in, discussing, and then, proposing a data 
set is definitely valuable. Yeah, but that’s the first step. You’ve done that part, you’ll get it. 
[Jamie Joyce]: Yeah, I’ve played some games. I love the painting games a lot, actually. 
That’s pretty fun. 

[Frode Hegland]: Very good. Just to answer your question about sit in. You’re all invited to 
just be in any of the meetings. It is the same time. It should be four to six UK time, but you 
Americans move the whole clock. So we follow you. Right now we will catch up with you. 
But sit in, say nothing, speak, whatever you feel like. It’s just a warm community. Mr, sorry, 


professor, not professor, Dr. Anderson. 


[Mark Anderson]: Okay. Well, thanks. I really enjoyed the presentation. And I’d love to 
hear someone just mentioning the idea of a data set because given the deconstruction 
you’re doing, there’s something really interesting there. At the moment, when people talk 
about data set, that just means “something I scraped out of an excel spreadsheet and shoved 
in a box and I now think it’s worth money”. Fundamentally not what I think data is. And it’s 
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really interesting too for this thought of VR, one of the things, so I’m kind of completely new, 
I suppose, two months, I guess, since I’ve looked at any certainly any mode in VR, and I’ve 
been using Oculus. And then, one of the things I’m really seeing is that lots of things that 
you’d think would work, just won’t. So take pretty much any 2D print visualisation you’ ve 
thought of, it’s not going to be instantly better for seeing it with an extra D. That’s for certain. 
Which is, in effect, why the data is so much more interesting. So rather than think, “How do I 
make this picture, this 2D picture appear in 3D?”. With all this richness, “how can I see 
things that I can’t show?”. Anyway, probably something that preaches to the converted there. 
Just a couple of thoughts but against you rather cheeky and I feel bad about it because they’ re 
only possible to make given the mass amount of work you’ve done and the wonderful 
deconstruction of the arguments. But I suppose the hypertext researcher are used to looking at 
non-linear paths and I was thinking, does your deconstruction process show you where the 
same sources or the same arguments occur in multiple parts in the graph? Because I think that 
becomes useful. And also, the whole ‘Johari window’ problem of the ‘unknown unknowns’. 
There is a danger and we’re all prone to it, is that, because it takes time to do this, by the time 
you’ve mapped everything out, that seems like the known world. And I can’t see how to still 
force myself to say, “Ah, that’s just the bit I know. Now let’s look at the broader thing” I 
don’t know if there’s an answer to that, but it’s an interesting challenge. And the one other 
thing I was thinking about prompted by the thought of changing views about how we trust 
data, sources, and things is, we seem to have moved into a world where there are massive 
first-mover advantages in being the first to complain, for instance. So there’s moral 
ascendancy being the first person to call the other person bad, regardless of the truth or 
situation. And it also tends I’ve been a thing creeping away if there are two pros and one con 
or vice versa, that’s seen as actually being an empirical measure of work. How do you cope 
without or does the deconstruction model not attempt that? Because I’m not saying it 
should, but what are your feelings on that side of things? 

[Jamie Joyce]: Yes, okay. So there are a couple of different things that you said and I’m 
going to try to remember and respond to them all. So, One—can the system understand that 
the sources are the same? Yes. But there’s no features built on top of that to make that easy. 
And it’s deceptively linear because we can actually copy and paste nodes all across and it 
does link all around it. And if we update one, it updates the other, etc. So it’s deceptively 
linear. But definitely, it could be more rich and useful as a knowledge graph if we build 
features to actually filter content like that. We don’t currently do that. The other thing that 
you mentioned is about knowns and unknowns. Actually, we’ve had extensive conversations 
at the Society Library about this very thing, because we have this technique called ‘Devil’s 


Advocacy’ research. It’s something we borrowed from the CIA. So, if you take a claim and 
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you just invert it to its opposite, and then, you go and try and steel man that, that’s a CIA 
technique at least as recent as 2009. And so, because we generate so many claims, I think our 
climate change database is like 396 thousand claims of a single expression, not variant 
phrases but single expressions. We could just invert all of those and have a whole other set. 
And then there’s, of course, you don’t just invert things in a binary sense, there’s all these 
shades of grey in between, there’s all these adjectives that you could add that slightly change 
the meaning of a claim. So there’s a lot that we could do there to, essentially, once we have 
the set that we have, the known, knowns, to invert, slightly adjust to drastically expand that. 
And then what we could show as visualisation and we thought about this, would it be a useful 
epistemic tool to show people? This is what we were able to steel man, but these are all the 
different research questions we have, that we were just able to generate that are relevant to a 
certain extent. It’s not just nonsense created by GPT-3, right? It could be relevant and we 
haven’t done the work yet. And we didn’t know if that would be that would increase 
intellectual humility and curiosity, or that would be really disincentivizing and discouraging. 
So it’s beyond our organisational capacity to that experiment. But we have been thinking 
about and are interested. When it comes to trust first movers, I see people coming into this 
space and being the first movers of complainers and I’m seeing them rise in popularity and 


it’s very interesting to watch. But I can’t remember why you brought that up. 


[Mark Anderson]: Well, it bleeds into this point about people getting overly empirical. So, 
“T’ve seen two supporting things. So clearly that’s more than one countervailing argument”. 
So, in other words, not just the user, the learner from this not actually learning, they actually 
have to evaluate. Having got to these sources and actually having to evaluate them. It’s not so 
much just counting up pros and cons. 

[Jamie Joyce]: Yes. I’m going to quickly see if I can pull this up really fast. So we’ve been 
thinking about that also. And one of the reasons why we have tags is to start qualifying things 
so we'll call out if an argument has no evidence since it’s just an opinion or logical fallacy, 
etc. Because we’re trying to combat some of those cognitive biases. And so one of the things 
that we want to do as clunky of an idea as this is, we do have an intro video where we’re 
going to try and prime people to not fall for these different cognitive biases. To tell them 
explicitly do not fall for this trick. Having more does not mean this is better argumentation or 
what have you, do not fall for this trick. We’re thinking about making it so that you can’t 
even unlock the paper map decision or library until you play a video that helps inoculate 
against that. And something that I’ve been just personally wondering is, can cognitive biases 
cancel each other out? So if people are one: more likely to remember the first thing they read, 
but they’re also more likely to remember things that are negative, should we always show the 


con positions or the no positions at the end? And should we, in these intro videos, tell them 
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that there’s no way that we can get around some of these biases? Because they’re hardwired 
in our brains. We’re just primed for them. So we’ve organised things in this way. If you think 
it’s biased, it’s because it is, but we’re trying to counteract this other bias. So we’re trying to 
find that, is there a communication medium whether asking them to watch a video or have a 
little character pop up, a little tiny robot librarian bloop up and be like, “Hey, just so you 
know, we did this for this reason because humans are biased and flawed and we’re just really 
trying to get you to enlightenment here”. We’re thinking about it. We have no great answer. 
And we do have a partner at Harvard and NYU who offered to run a polarisation study to see 
if the way that we map content can depolarise attitude. So we are interested in partnering with 
universities to really rigorously test some of the features that we’re thinking about, just to see 
if it does have a pro-social positive impact because we’re not interested in persuading 
anyone. We’re not interested in driving anyone towards any conclusions. We just have the 
librarian goal of enabling enlightenment through access to information. And for us, 
enlightenment means potentially open mind through depolarised attitudes, inoculation against 
disinformation, intellectual humility, increased subject matter knowledge and increased 
comprehension of complexity. So just overall more curiosity, open-mindedness, and 
comprehension without being inflicted by bad attitudes, depolarisation, disinformation, and 
things like that. 


[Mark Anderson]: Well. that’s lovely to hear, actually. And I’m thinking, of course, that 
again, the joy of you having such a deep and rich data set is, for instance, although, you can’t 
necessarily answer some of the stuff on these biases, there’s a lovely substrate for someone to 
work on. I mean this is again where I think people fail to see where the real value in the data 
is. It’s not like you’re going to sell this to somebody. It’s the fact that it’s just hours of 
dedicated work. And especially doing it from, in a sense a neutral, for a want of a better 
word, but a standpoint which tremendously important because you rightly stay. I mean if 
you’ve got some bias in there or if you’ve got more than a trivial amount of bias in there to 
start with, then you’re building on sand. And just because I see hands up, but one final 
thought is, when you mention the fact, yes, inputs to turn up across the piece. That, for 
instance, might be an area where having extra dimensions visualisation might be exploitable 
because it’s really hard to do on a flat surface because the worst thing is all you said end up 
with lines all over the place, and it’s alternatively complicated. But I think that one of the 
things that are submerging in our exploration of what VR is its ability to, you don’t 
necessarily have to remove things, it’s reducing the salience of some things. Bringing it, 
dialling it upon others. So it’s all there. It’s all somewhere in the space, perhaps. But what 
you’re seeing is the connection that’s pertinent at the time. It’s a different sort of interaction. 


You will ask for the thing you’re interested in knowing and bring it forward. Of course, 
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thinking that and making that up is the journey we’re on at the moment. But, thanks. I find 


that really interesting. 


[Jamie Joyce]: Thanks for your questions, Mark. 


[Daveed Benjamin]: Excellent. Hey, Jamie. So my question is, well, the premise of it, is that 
what you’re producing is going to be incredibly valuable, I’m just making that assumption. 
And I’m also looking at it and just seeing it, seems like there’s just such a tremendous 
amount of work that goes into just one inquiry. And what I’m wondering is: What does it 
take right now, for example, to do something like the California Nuclear Plant both in 
kind of human resources, as well as elapsed time? And then I’m also wondering what do 
you foresee in the future, in terms of being able to streamline that with both automation and 
potentially AI? What do you what are you shooting for, in terms of human resources and 
elapsed time? And then, the third part of that is: Is decentralisation, at all, on your radar in 
that possibility of bringing in a much larger group of analysts to do certain pieces of the 
work? 

[Jamie Joyce]: Yep. Great set of questions. So I believe the Diablo Canyon Power Plant 
project is about 10 weeks old. We got two more weeks left to wrap it all up. And that was 
inventing the paper visualisation along the way. We had four full-time analysts. I was a part- 
time analyst. So it’s kind of a small team in a relatively quick period of time and I think we 
owe a lot of that to the tools we built ahead of time in the past and our methodology. And the 
fact that we have years of experience training students through our educational curriculum, so 
we know how to train people to like quickly understand debate map, quickly understand what 
we might claim, use the tools to find content, but I honestly think, a lot of the different tasks, 
not the work, I think we’re going to be working with librarians and human analysts for a long 
time, but I think a lot of the tasks, discrete tasks, can be automated. We’re tinkering with 
some of those right now. I’m fundraising for some of that right now. I’ve got a lot of ideas 
about what’s possible in both like claim mining, syllogism generation, mass deconstruction, 
there’s a whole bunch of ideas that I have. And there’s already tools that exist that we could 
be experimenting with more. So I’m excited about that. You mentioned decentralisation, I 
think there’s a question between that. With decentralisation, the thing is that, language is so 
flexible and dense, and some people are not very precise in their expression. So it depends 
upon the knowledge that you are working with, first of all, because we’re working across 
different media types, there’s a lot of flexibility in that language, there’s a lot of ways in 
which people can misinterpret, they can imply, they can bias the interpretation. So if we were 
to welcome more of a crowd, there would be discreet tasks that I would allocate to them. But 


I would not trust a crowd to be responsible for the emerging structuring of a deliberation. And 
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that’s because, unless this entire crowd is somehow really well trained in understanding what 
is relevant argumentation and what is not relevant argumentation, you’re going to end up 
with a humongous spaghettified mess. If you look at existing platforms, I’ve looked at a lot 
of platforms, if you look at existing platforms, you’ll notice that the argumentation is either 
very vague enough, where a lot of the relevance can be applied. Or it’s not really fine enough 
in terms of actually establishing the logical relationship between the points even if the points 
are more specific. So it’s not to the level of rigour that we’re interested in the Society Library. 
And that’s because like the knowledge project products that we’re looking to create, even 
though we’re creating the options for people to simplify things. Simplify this and put it into 
simple variant phrasing, for example. Just give me the gist of it. We want to give people that 
option. We actually want to do, as rigorous work as we possibly can, in terms of 
deconstructing arguments into their processes and conclusions. Because if I feel like if you 
don’t do that then it’s always going to yield more and more argumentation because people 
will misunderstand what’s implied. So, if you actually pull apart the argument like, “This is 
every single stage of what we mean. This is all the data that supports those things”, maybe it 
allows some tiny small subsection of readers to really appreciate that more. A lot of people 
are not going to want that level of detail. Just give me the gist so I can see and make my 
decision. So there are certain things I think the crowd could do really well. I think archiving 
is something the crowd could do really well. I think tagging is something the crowd could do 
really well. Modelling argumentation I think that’s a really high skilled skill. I think that’s a 
really technical skill and I wouldn’t trust like a hundred thousand people to do that in a 
meaningful way. I already get in wiki wars on Wikipedia, for example, and that’s just an 
encyclopaedia page and there are no rigorous rules about the relationship between sentences 
and Wikipedia. And yet, people still fight about that. So yeah, that’s my point on that. 


[IN CHAT) From Marc-Antoine Parent]: That does not mean that it cannot be partly 
crowdsourced in principle, but certainly not naively 

[Jamie Joyce]: Yeah, partly crowdsourced in principle. That’s right. I agree. There are parts 
of it that could be crowdsourced like finding the topics, getting the resources, finding how 
topics, and base arguments appear in certain resources. So again, archiving and tagging, I 
think it was a great crowd job. But modelling, I think requires a lot of skill and a lot of 
editorial review. I review the work of all of the analysts. We review each other’s work. In the 
future I want us to have more of an inner coded system, where a lot of the work is actually 
redundantly performed, the same people performing the exact same task so we can actually 
see the difference, and see if that difference is statistically significant. There are people who 


build distributed content analysis platforms that I really like. They’re friends of ours, they 
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collaborate with us on certain things. I’m not yet finished with fine-tuning our method 
enough to know what we want to have as a part of distributed content analysis and what can 
be automated. So maybe a few moderations down we’ll have the right combination of like, 
“Okay, we’re going to hard code this modelling into something that is distributed”, and then 
also have AI help us with certain discrete tasks, and maybe a crowd. We’ve been poked and 


provoked to do a DAO, as well. So, I don’t know if we will, but. 


[Daveed Benjamin]: The question that that was in between was actually related to the first 
question. In the best of all worlds, where do you see the elapsed time getting to... Because, 
especially, when we’re talking about a culture with this first complaint dynamic happening. 
It’s like getting this information out quickly, I think could be really valuable. 

[Jamie Joyce]: Yeah I like to think and try to orient our work towards constantly imagining it 
being possible instantaneously. There being constant monitoring, construction, and modelling 
that’s happening, I think we’re really far away from that. But that’s what I would like to get 
to. Essentially, the Society Library, being a large enough institute to have the manpower to 
respond where we need human analysts intervening, and also the technology to be observing, 
deconstructing, labelling, and doing base categorisations. De-duplicating these sorts of 
things. Finding the right combination, a lot of the unloaded work is being done by AI, and we 
have enough staff, librarians, essentially, serving society quickly, modelling up this content, 
where just absolute elite experts, and then, having all the tools that they need, in order to 
quickly do that. So, journalists are reporting and stuff is happening on TV we can be quick on 
incorporating that into higher dimensions and more complex mapping, epidemic mapping of 
a situation. So I’m hoping one day, I don’t know if itll be in my lifetime, I’m probably 
underestimating technology, but I don’t always imagine that it’s in my lifetime, but I’m 
angling towards us having an instantaneous institution for this. At least for publicly 
accessible knowledge. 

[Frode Hegland]: I see Fabien has his hand up, but I just want to say thank you for 
something very specific. And that is surfing the line between being popularist and being 
arrogant, or elitist. Because I do think it is really important to still value expertise, and our 
current culture isn’t so happy with that. I’m sure you have seen ‘Hamilton’. I’m sure you 
remember Aaron Burr, I could have a beer with him, right? This is horrendous damage that is 
being dealt to us. So by you standing up for expertise without being arrogant, without taking 
a position, that’s just fantastic. So, thank you for that. And, Fabien? 

[Fabien Benetou]: Yeah, my question is, and maybe I missed it, but how do you interact 
with the result? Or how does somebody who wants to learn, let’s say, get the expertise out of 
a topic, gets that? What I saw through the presentation, and again, maybe I misunderstood, 


but was unfolding the different part of the map. But is there another way to interact? I’m 
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asking this specifically to see also again how could this eventually be coded or considered for 
VR? Because a visualisation, of course, is an object and you can see it but you can do more 
with it. You can, for example, fold and unfold, but once you got, again for VR, their 
controllers, or even your hands, you can manipulate it. Or if it’s text, you can copy-paste it. 
So they are very different and rich interactions. So, yeah. I’m wondering what’s the styles of 
interactions right now? And what was the thinking process behind it? Because more 
interaction doesn’t necessarily mean better. You want to concentrate on something 


productive. So, yeah. If you could dig a bit more there, I’d appreciate it. 


[Jamie Joyce]: Yep there’s a couple of different visualisations that we currently offer. It’s 
essentially the same data, but we compress it and organise it differently. For some 
visualisations, we filter things out. One thing that we do is we do make the map available. 
People can un-click and expand and do all that. I don’t think a lot of people are going to find 
that attractive. Some people are going to love it. Two is, we’re revealing something called 
Society Library Papers, where all of the data in that debate map is actually structured in a 
piece of paper, where you can click on the lines and it opens up options to further unfold, not 
only the argumentation but into the note itself. You can refresh so the different phrases and 
different ways of expressing things show up. You can press keys that will swiftly change it 
from technical to simple language. So you’Il have your standard way it’s expressed, you can 
flip between those like, “Show me the more technical. Show me the more simple”. It unpacks 
the argumentation. So, Papers is the new thing that we’re rolling out. The other thing that we 
do is create decision-making models. So we just zero in on one sub-section of argumentation 
at a time, and people can add values to how they’re weighing the different arguments. And 
then, essentially, we ask them to micro vote. Where is the strongest argumentation in this one 
subset? And then they move on to the next set, and they move on to the next set. At the end, 
it’s shown to them like, “Okay, well. Where do you stand on these issues? Economic, 
environmental, etc.”. The decision-making model is the thing that we do at City Council. And 
then the other thing we do is we just make all the resources that we generated available in a 
tech searchable library. So you can look through all the references that have been included in 
this data set, you can keyword search and find the claims not necessarily associated with each 
other. You can search it on the map as well, so it’Il bring you to the part of the map where that 
claim is. And that claim can be in multiple places, so you can bounce around and see where 
all the places that this claim is. Or you can just search it in a library list. So those are the four 
things that we have, in terms of visualising content now, but I’m excited about, thinking 
about doing the same, kind of, paper unpacking and unfolding, but with video. And then 
unless we do something specific, like write legislation. We also have a designer who’s 


working on a phone app version of it. In terms of being able to interact, for example, with the 
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paper, you can click on any line and it gives you the option to expand it in various 
dimensions. Show it to me in a video. Show it to me in a podcast where the references that, 
support this, where the bearing phrase is, etc. It would be so cool if in VR, and again, I don’t 
know if this would actually translate well. It’d be cool if you could take a statement and 
actually open it up. Grab that statement, open it up, “Oh, I can see all the videos, TV clips, 
where this occurred”. Swipe to the left, okay, here are the references. Okay, definitions. Okay, 
media artefacts. Okay, close that one up, star that, want to look at that later, grab the next one, 
let’s open that up and take a look at what that looks like, or do a motion like this, and it just 
spills out all the different claims that support it. I think there could be a lot of cool interactive 
ways that could make text a little less boring, simply because you’re moving your body. And 
moving your body may create endorphins, and make you a little bit more happy and excited 
about stuff. And that’s what we’re trying to do with Papers. With Papers, I didn’t show you 
any interactivity at all, I just showed you the mock-up. But we’re really focused on how it 
feels. The slickness of the unpacking. The slight little sounds. We’ve been looking at the 
colours when rendered to accommodate for different like visual, I don’t want to say 
impairments, but different visual differences. And it looks gorgeous, all of them, in my 
opinion. So we’ve been really focusing on the feel of it. Because we know it’s limited by text, 
but if we could translate that to VR, I think, it could be much more interesting just by being 
able to literally work with knowledge. Grab knowledge, put that over there, unpack, all that 
stuff. 


[Fabien Benetou]: A quick remark on this, I don’t know if you’re familiar with the 
interactive explorations. Basically it’s the idea that you can have exercises in the middle of a 
piece. You don’t just have a piece of text but, you have an exercise. And it’s not just a 
textbook exercise, but it’s part of a story. So that you, in order to get the core idea of this 
article or concept or paper, you go through an exercise. So it’s a guided interaction, basically. 
You’re not just freely moving things around, but you’re solving a small challenge that’s going 
to get you to this eureka moment that the author of the paper had at some point and that’s 
why they were sharing this some kind of information. And I think that, also, is something that 
could be valuable. Of course, freely manipulating, but a guided manipulation that makes the 
person, who has to interact with, get the point of it. It could also be quite interesting, I think. 
Here, I don’t know how literal or metaphorical you want to be. Let’s say, you display a 
nuclear power plant. How close you are. You can zoom in and out to the atomic level or not. 
Yeah, there is a lot that can be done there. 

[Jamie Joyce]: Oh, sorry. I didn’t mean to interrupt you if you saw my hand waving, because 
I got really excited. That’s good because we did a mock-up once. We didn’t get enough 


money for this project. But we actually did that for our Covid collection. So we built a 3D 
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lab. We just did this in Prezi so it was a very superficial mock-up. We did it in Prezi because 
it has that zooming in and out feature. And we built a 3D lab where you could essentially 
zoom into the lungs of the lab worker who was there and explore all the different 
argumentation about respiratory health, and what Covid does to the body. You could zoom 
into the microscope to learn about argumentation about what SARS-COV? as a virus is, what 
are its features, what are the pictures that have been grabbed, what kind of telescope, etc. We 
did that, where we built a scene and people could zoom into different dimensions of it, to 
explore different topics, sub-topics of debate within that. But we didn’t get enough funding to 
really do for real. But I think that would be really fun. 


[Fabien Benetou]: The funding part, to be honest, it’s about to lack in the sense that, it’s 
super demanding to get this kind of materials. Designing a 3D experience. But I think for 
those two cases, nuclear or Covid, they are excellent for it, because there are skills that are 
not graspable for most of us. And I think for a technical expert then it becomes natural 
because you did the exercise so many times that it does become natural. But being able to 
change scales, in a way that still makes the intangible, tangible, it’s a perfect use case for this. 
[Jamie Joyce]: Amazing. I’m excited to sit on all these meetings. 


[Frode Hegland]: Peter are you is that an earlier hand or is that a fresh hand? 


[Peter Wasilko]: It’s an ongoing hand that’s been having things added to the queue. Your 
description of waving your hands around reminds me a little bit of the user interface depicted 
minority report. That was a very interesting movie to take a look at for the VR-type 
visualisation. Well, more AR-style visualisations in it. Also the talk about instant votes and 
things remind me of a wonderful episode of The Orville called ‘Majority Rule’ where 
everyone would walk around with smart badges that had an up arrow and a down arrow. And 
if you accumulated too many down arrow votes, you’! be lobotomised by the society. So, 
just fascinating. Then substantively, have you taken a look at the foresight literature? There’s 
a concept called ‘Futures Cone’, and I put a link for that in the chat. The basic idea is to 
represent multiple possible futures. And it seems like that could be a good visualisation for 
providing an organisational access layer to the dialogue, because some of the different points 
would correspond to the same possible future. And that could provide a different view on the 
evolving debate structure. So, for instance, there would be one possible future where the 
claims that the climate temperature is going to go out of control is correct. Then there’s 
another possible future where it turns out that those studies might not necessarily have been 
accurate, more an artefact of modelling software. So you could take those possible futures 
and represent them in a visualisation, and use that as a filter onto the debate structure. Also, 


another very interesting diagramming technique is the use of state machines. Very popular in 
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computer science. And it’s been touched on in linguistics and parsing to some extent. And the 
notion there is that you have a series of states, with transitions between states, and you could 
associate different arguments with transitions between states, where the states could represent 
possible futures in the futures cone visualisation. Also, I would suggest having a look at the 
system dynamics literature, which has its own suite of visualisations. Some of which are very 
nice web-based tools that look at feedback loops between different stocks and flows. And that 
again could provide a filter into the diagram structure as some of the diagrams would relate to 
different elements in that visualisation. So those are a couple of possible filtering access 
layers that you might want to have a look at. I have all the links to the side chat. 


[Jamie Joyce]: Thank you so much. I think I found everything that you post in the side chat. 
I super appreciate that. Getting some interaction interfacing with the forecasting community 
has been of interest. But I haven’t had the capacity to explore what is the extent to which they 
model futures. Because I obviously would be interested in taking those projections, 
deconstructing it and seeing what it looks like, in terms of, translating it to Society Library 
language and concepts, so I can get a handle on it. But I haven’t had the capacity to check 
that out. But thank you for these links, because it’s definitely on my list. 


[Frode Hegland]: Talking of links. First of all, as you know, this will be transcribed and put 
into the journal, both your presentation, this dialogue, and the chat log. So whoever’s 
interested, go and have a look at our first two issues. Mark and I are still learning how to 
make it navigable because there’s a lot in there. But then my question is actually quite 
different and that is, a few years ago I took an online quiz in a Norwegian newspaper on 
politics, and it asked me, what do you think of this, what do you think of that? Click through, 
very simple. At the end of it, turned out that I should be voting for the Christian Democrats. 
Which was a huge surprise because, whatever. But so my question to you then is: Have you 
considered letting your users model themselves in such a way, so that when they go into 
this, they have a stated position that the system understands that is based on their 
answers? 

[Jamie Joyce]: I mean, when I think about those kinds of quizzes, the first thing that pops 
into my mind is, I really want to see the data that they’re using to suggest that. Because I 
want to know if their interpretation of this candidate’s position is the same as the language I 
would use to describe my position, interpreted by the language that they’re giving me, maybe 
a multi-choice format to express it. So I personally have trust issues with those particular 
things. There are so many different possibilities to create products from these data sets. 
[Frode Hegland]: I’m thinking not necessarily about the quiz, because, yes, that may just 


have been a journalistic gimmick. I’m thinking more about coming in and let’s say, most of 
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us here, we go and say environment concern, high. We state where we are on specific issues. 
Health care should be shared or not? Just a few yes or no. Because that’!l put you because 
you talked about a constellation earlier, there’s also the thing, the spider graph, where you 
have lots of different dimensions and you can then see the shape of different people. But as 
long as people can be shown that dimension of themselves and say, “this is who I believe I 
am”, that may help their interaction, somehow. I don’t know if I’m going on a huge tangent 
or not. 

[Jamie Joyce]: I mean, one thing I think is interesting is that, these types of quizzes, whether 
it’s Enneagram, or Myers-Briggst, or whatever, people love to conceptualise themselves, I 
think. These things are popular because, within a certain subset of people, they want to call 
themselves and identify themselves as something. Just being like, “Oh, I’m INTJ. Or ?’'m 
‘this’ or ‘that’.”. Some people really love that stuff. I think that could be a cool offering in 
terms of, “Here is your graph of beliefs”. Like an astrological chart. Here’s your sun sign and 
whatever sign. Here’s the dimensionality of your beliefs. I think that could be cool. But also, 
you made me remember something, which is, when we were talking earlier about trying to 
combat cognitive biases, and I mentioned that I believe it’s the cognitive bias that people 
remember the first thing that they see. Or people may have a backfire effect if they see 
something that immediately contradicts what they already believe. So people taking a quiz 
upon being introduced to a new subject, it would require for us to have an account system. 
They’d have to create an account, so we can remember these preferences. I think it could be, 
potentially, a way to combat bias if we knew what people strongly held beliefs are so that 
those could be expressed first, and then, they can be confirmed as being understood. Because 
I think that’s how you can overcome backfire, is you let people know you model to them who 
they are, we hear you, we understand you, and here’s the strongest version of the thing that 
you believe, here’s everything that you could possibly want, and now let’s go explore 
everything else. So I think that would be useful. Again, to start changing the way we interact 
with information, to enable enlightenment and open-mindedness for people who want to opt 
into something like that. 

[Frode Hegland]: Yeah, I think that’s very interesting what you’re saying because a 
cognitive bias is not a bad thing, in and of itself. Same as prejudice. Also, is not a bad thing, 
in and of itself. Without them, we can’t function in the world. And, of course, who you 
express you are, depends on the circumstances where you are asked to show who you are. I 
do think it is really an important issue because, for instance, our son, beautiful four and a 
half-year-old, Edgar, he goes to a Catholic school. But we weren’t sure about putting him in a 
religious system. But the reason we decided to do that was, I’d much rather argue morals with 


him at home, rather than him go to a school that doesn’t allow that. And then, try to teach him 
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to be nice, because a lot of these decisions come down to how do you see your neighbours, 
how do you see yourself, how do you see the planet, all that stuff. Everything is filtered. You 
can’t argue facts is something we keep being told again, and again. So I’m just wondering, 
you have this incredible information landscape that is intelligently put together, if there was a 
mechanism of someone, maybe even stating their beliefs, and then when they go into the 
system finding out that they’re not actually behaving within their stated beliefs. The typical 
thing being a right-wing Christian. There’s no chance in hell Jesus was a right-wing 
Christian. As an example. To not only get the information in, but having a thing that is 
representing where it goes in. 

[Jamie Joyce]: I think in the future we’re going to have a lot more capabilities to do things 
like that. And I hope so because knowledge and information is so powerful and impactful, 
and if we could just improve that relationship or objectify knowledge and have a new type of 
etiquette around how we interact with it, and allowing it to change us, and open up us. And 
mirroring that it understands us and can, again, contrast us. I think that could be really 
wonderful. I think it’s far out for the level of sophistication that you’re talking about, or I’m 
just failing to imagine. But I hope we get there, because I think that would just be so lovely. I 
personally love to think of humanity as a species on an information diet. I think how we 
really survive is on information. All the different inputs that we have. I think making that 
relationship even more sophisticated is, hopefully, in our future and for the best. 

[Frode Hegland]: Yeah, that’s wonderful. And sorry, as a just a tiny little thing, and that’s, if 
you read Jaron Lanier’s booky+ on his VR experience journey, it’s very little about the 
environment and very much about the self. How you change yourself in the environment. So 
to think about this, in this context of being able to go into this information with an awareness 
of different ways you are yourself in this space. I know we’re talking a huge down-the-line 


kind of thing, but it was just interesting to hear that. Right, Mark. I will shut up for a minute. 


[Mark Anderson]: Okay. Very quickly because I see other hands are up. And just to restrict 
myself to just one observation. Another thing I think is interesting that comes out of the really 
interesting deep landscape, data landscape you’re making, is the ability to look at the, well, 
almost the meta-metadata. So when you look across the problem space, where are the 
references coming from? So, in other words, there’s a whole skein that goes over the top of 
this, which is not part of the augmentation or argument discovery, per se. That’s quite useful 
in an intelligent, very small ‘i’ sense, in terms of, understanding the problem environment 
space, I think. Anyway, I'll leave it there because I see there are some people, and some 


haven’t spoken yet. 


[Jamie Joyce]: I will just say quickly, Mark, what you may love to hear is that we do take 
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great care to steel man things. So if we find arguments on TV or in news, we will try to see 
where is the rigorous academic literature on this. So it’s not just by luck that we’re 
identifying and associating arguments with media types, because we always try to get the 
most rigorous as possible, and the most accessible. So if it exists in different media sets, 
we’re really looking for them. 

[Mark Anderson]: Yeah, it’s just this interesting thing that sometimes now, it seem certain 
sorts of arguments seem to come from a... Or in a certain type, I don’t want to typify it too 
much because then you get into the labelling, but it’s just the sentence that, whereas you 
might think it’d be distributed across the piece, going through all channels, or all age groups, 
or whatever: it can be quite fragmented. And that’s the kind of thing that the rich data that 
you’re collecting also enables you to see, I think. 


[Jamie Joyce]: Yep, I agree. Okay, who would like to go next? Karl or Peter? 


[Karl Hebenstreit Jr]: Yeah. I posted a link on, Peter Elbow had this article about the 
‘Believing Game’ and it’s some nice connection between that and the ‘Six Thinking Hats’. So 
it’s systematically speaking validity in what you don’t agree with. And then, it’s interesting 
with dialogue mapping and Jeff Conklin’s. Both Jeff and Edward de Bono, they focused so 
much on dialogue mapping and Six Thinking Hats being a meeting facilitation process. But 
then, there’s also the whole individual sense-making process too. I’m very big into dialogue 
and the facilitation process. How do we get people engaging in real-time? A thought I had 
to bring that to your attention. 

[Jamie Joyce]: Yeah, thank you. I copied those I’ve never heard of the Believing Game or 
the Six Thinking Hats, actually. And I think that there’s a lot that we could learn from 
bridging communities and facilitation communities. Because what we’re trying to do is a 
technologically induce a space where people can interact with knowledge maybe as if and the 
different positions maybe as if they were interacting with a person. We’re not simulating that. 
But like what are some things, in terms of, the visualisation itself that could create that 
container? That would make people feel receptive and that sort of thing? So I think there’s a 
lot that we can learn, and I try to pick up things here and there from facilitation, mediation, 


and bridging, to learn those things. 


[Marc-Antoine Parent]: Just a quick thing. I mean, Conklin—I’m working with Jeff 
Conklin right now. His work really shows the value of facilitators in de-personalising 
arguments and creating these syntheses, usually in real-time. And I think it goes with what we 
were saying earlier about the importance of argumentation as a skill. And this was synthesis 


map-making and consensus making as a skill. And, yes. I agree totally. The question is how 
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to weave individual sense-making, which is a more and more important activity, into 
creating these synthesis maps? And the question of creating synthesis from individual 
curated maps to collective curated maps is really the key articulation. But it’s not going to be 
just crowdsourcing, it has to be learned. And there are many paths towards learning to do 


that. And it has to be social learning about how to create these consensus maps. 


[Karl Hebenstreit Jr]: Just one quick thing too with the way Jeff separated out. So you have 
the issue mapping, which is gaining the competency with compendium and creating the 
maps. And then there’s the dialogue mapping, which is the facilitation process. I think that’s 
really important for all these tools. 

[Marc-Antoine Parent]: I personally believe, sorry, Jamie, we will need more than one 
mapping when we will need to connect them. For example, I think what Jamie is doing is a 
wonderful epistemic map. Why do people believe this? And someone was bringing these 
future maps. But when you do a future map it’s, this may lead to that, this may lead to 

that. It’s a totally different temporal presentation. They shouldn’t be on the same map. But 
you would want to know why do you believe that this may lead to that, which is the epistemic 
dimension. Connected to that and vice versa, right? Why is the belief that this may lead to 
that also feeding into the epistemic questions? They’re different representations. I don’t think 
there’s ever going to be one representation to them all, but we need to make a representation 
(indistinct). 

[Jamie Joyce]: Speaking of communication. One of the things that I was thinking too is that 
I’m not a facilitator. So I have very limited knowledge of what that tails. But my 
understanding is, facilitation and mediation include deploying all sorts of different 
communication techniques to position people in the space where they can then proceed with a 
conversation and interact with something that’s potentially conflictual. So I’ve been thinking 
too is, maybe there could be, in thinking of how do we borrow from facilitation to enable 
interaction with our content in a successful way, it may there be a chat bot feature where we 
turn it on different facilitation communication strategies. So someone’s interacting with 
knowledge. And someone’s going off track, they know how to like, “Oh, hey. Okay. Let me 
reframe what you just said, and let’s move it now over back to the map”. So there’d be a 
relationship between a chat bot that could carry on the conversational AI element, to walk 
people through the epistemic map, because it may be too dry to ask someone to go back and 
forth, pro and con, down from a position to more precise argumentation. That may be too 
much. Besides readers who are just generally interested in exploring a deliberation. But in 
deploying it to a chat bot, integrating it with that, I think is something that could be in the 
future, as well. And if someone isn’t already working on capturing all of the facilitation 
techniques, I hope they do and they train an AI to be having that, would be cool. 
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[Frode Hegland]: Yeah. You’re talking about something being too cold. I think that’s a very 
good question here. You’re building an incredible intellectual tool and of course, emotions 
will come into it at some point. So you’re enthusiastic for VR and I think you understand VR 
like we do. Just multi-dimensional. Doesn’t matter if it’s a headset, or whatever particular. 
Will definitely come in to make people feel more embodied, more involved, and more aware. 
One of the great things about this bookyt I keep holding up is, it points out that, if you want to 
be more rational, you listen to your body more than your head. Your head is more emotional 
than your body. Which was a bit of a surprise. So, we can help people get those 
understandings. I’m not sure why Brandel is not here. He’s always available. He’s very 
involved, so there must be a good reason. So when he watches the video of it at some point, 
we miss you, Brandel. I hope to see you together with the group soon. We have seven more 


minutes. And Peter will have one of the final questions. 


[Peter Wasilko]: All right. I was wondering whether you’re doing anything to flag bias of 
the sources that are behind the sources that are being referenced in the articles? A very 
common phenomena is that you’Il have some group with the name like, ‘Concerned parents 
trying to improve safety in schools’ and etc. Then you look at it, and you find out that it’s 
really a front group for gun manufacturers. Or you’ll have a piece of legislation and the name 
of the legislation is the exact inverse of what the functional result of the legislation would be. 
Also, I was wondering whether you’ re looking at pop culture as sources of argumentation 
too. There’s a wonderful wiki called TV Tropes that has links to just about every single 
movie, book, manga piece of resource out there. Plus they also cross-link examples of those 
probes in real life. So, you can find whole sections on every TV show that discuss climate 
change in there. And sometimes, you’ll actually have people use the fictional medium to 
express policy diagram disputes because they’ re afraid that if they put it out on Twitter, they 
might get cancelled from Twitter, but you can have an alien of some crazy race make the 
argument in a science fiction story, and you can discuss those social and policy issues that 
you couldn’t otherwise. The Orville is a great source of those sorts of stories. 

[Jamie Joyce]: First of all, I just want to say, Peter, oh, my god. Thank you so much. I’m so 
excited for TV Tropes. Thank you. We deconstruct film, but not television shows and non- 
documentary films. So this actually just may be a whole other source of media that we may 
really dive into, because I mean, we do archive memes, and we make memes available. So 
when you said pop culture, I was like, “Oh, yeah. We do the graphic image memes, for sure”. 
But I didn’t think about, yeah, an alien on a sci-fi film making a critique about something. 
Didn’t think of that. Thank you very much, Peter. And then as for your other question. Yes. 


But we have to be careful with labelling. Because labelling is very much a matter of fact and 
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we don’t want to make a mistake and be incorrect about matters of fact. So when we label 
something opinion, or needs to be fact-checked, or no evidence provided, or this is cherry- 
picking or something like that, it’s because it is very easily associated with matters of fact. 
There’s a definition for a thing that’s commonly accepted. This meets a definition (indistinct). 
But when it comes to astroturfing, predatory journals and things like that, it’s more of a 
matter of argumentation. And so, we actually do model that in argumentation. So, for 
example, there’s this whole set of content from this one person who has published only in 
predatory journals, and we had to deconstruct the website and essentially build out all the 
argumentation about how the data is not verifiable, the journal in which it was published, it is 
not credible in these ways. But we had to model that as argumentation. And what we have to 
do, in terms of our responsibility, is just, we have to make sure when it’s visualised, they pop 
up at the same time. So it’s not buried, all the counter argumentation that suggests this is 
invalid it’s, “Hey, because of the severity and weight of the argumentations, they get this 
other data, you need to see these things at the same time”. So, yeah. We do that. We do our 
due diligence and look on those lists of suggested predatory journals. And then we check out 
the website. Did the website, essentially, say it’s pay-to-publish? Do they have no peer 
review? Do they have no editorial board? Essentially pick it apart. And then, make sure that, 


that metadata express through argumentation. Essentially saying this is invalid or, you know... 


[Peter Wasilko]: And it also can be subtle, for this person might have a gorgeous new office 
in the Pfizer wing of his school and be publishing strong arguments in favour of Pfizer’s 
latest designer drug. And then, you’d wonder if he did a study that was designed to be able to 
find problems with that drug, would his school still have the funding to build that building? 
Or would he suddenly find that the primary source of his salary has gone away? And that 
might be influencing the way he structures his science. There’s also a community, I think it’s 
called ‘Tea’t, that’s looking at reproducibility in science, which you might want to have a 
look at, if you aren’t already familiar with that work. 

[Jamie Joyce]: There’s a whole bunch of people working on the reproducibility issue. But 
what are they called? 

[Peter Wasilko]: ‘Tea’, I think is their acronym. 

[Frode Hegland]: Since we started a bit late, we will do a few more minutes. And, actually, 


Fabien, you go first. 


[Fabien Benetou]: Just to bounce back, I also posted on the chat a book on agnotologyt and 
the study of political ignorance. I really warmly recommend it. I say warmly even though it’s 
a horrible thing. But I think it is important. It’s very exciting to hear the process you’re 
going through. And that prompted me to wonder. So, what I did at some point was, I 
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gathered most of the links of the articles whatever random pages I read. And I put them 
on my wiki. And what I did was the opposite. Meaning that I have a plugin from the 
browser that says, “Oh, you’ve already read that page, and that page is on that topic”. 
So I can browse back to my own notes based on what I’m seeing on my browser, on my 
screen right now. And I’m wondering if that could, also, be a way, because it sounds like your 
process is very thorough and could be practical behind, quote-unquote, just the map itself. 
But on your normal browsing session, being able to connect back to the map, for example. So 
I’m wondering if you’ve done that? Or if you believe that could be useful to browse the web? 
And then, as you go through a document, a piece of information that you already analysed, 
and it’s already referenced, if we could link back and browse the map at a certain point? I'll 
do a bit of promotion for the book I put in the chat, in the meantime. If you’re wondering 
about the tobacco industry, alcohol industry, and oil industry too. Different sales(?) of Nobel 
prizes on topics that we’re not necessarily familiar with. And that you see the same heads, 
that’s a great book. That’s an extremely sad one, but you see the history of convincing people 
that don’t actually have the expertise but still have the intellectual prestige. So, yeah. Very 
valuable. 

[Jamie Joyce]: Okay. So, yeah. Short answer, yes, we’ve been thinking about that, Fabien. 
One of the new tools that we want to build, actually, is going to be a web annotation system. 
So that people can go all the way through the chain of us collecting content, extracting it. You 
saw the spreadsheets where we copy things over to text and then deconstruct from there, 
that’s really lovely because if linked to those things people can see the exact line. But it 
would be lovely if it was just native. They can go to the archive.org website and see we’ve 
highlighted this and here is. And then, having that as a plugin that other people can go and 
traverse and just opt in to see, “Has the Society Library pulled this out?” And seen this 
somewhere, and implemented this somewhere, is definitely in the timeline of things that we 
want to incorporate, for sure. Not possibly yet, but yes, we’ve been thinking about it. And 
here’s the other thing too. I don’t have a huge amount of hope for integrating with Twitter and 
Facebook and things like that, because I know a lot of friends of mine, who created great 
products have not been able to get through the door. But that could be a thing too, also, let’s 
say, if there was a way of detecting the semantic similarity of a tweet with a Society Library 
snippet. They do this with Wikipedia on YouTube for example. There’s a particular topic with 
the, “Here’s the Wikipedia page on it”, in like a bar underneath, to try and promote people 
going to a source that YouTube finds to be substantial, in order to look and research into that 
more. And then, it could also be such that, journalists could reference our databases, and 
essentially, cite it in their news articles and people could link out to go see the Library. So it’s 


in the creation of new content, it connects to the Library, and then, the Library can also 
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backtrack to, this is where we derive the claims that populate this library in other media 


content as well. And that’s just web annotation of plugins and things like that. 


[Frode Hegland]: So final for me, anyway, is the work that I’m doing with my basic 
software, Author and Reader it’s bizarre. It took me two hours to see how it kind of connected 
with this. I think it’s probably because what I’m doing is so insanely much simpler than this. 
But the reason I want to highlight it to ask you a question is, in Author, which is very much 
targeted at students, part of the writing is for them to define things. If it’s important for them, 
write what it is. So you would write something like, Doug Engelbart and then a definition. In 
my case I would write, Doug Engelbart was my friend blah, blah, blah. So it’s personal. It’s 
not pretending to be objective truth. And then, maybe I’1l mention SRI. If I then somewhere 
else write SRI, when I then go back to the map, and I move anything I want, click on Doug 
there will be a line to SRI, purely because the text mentions it. Nothing fancier. But the 
reason I’m highlighting it to you is that you have this incredibly rich environment, it’s simply 
that, it seems that, if you make people define things, it helps their own thinking. And if 
they can then see how they connect. So I’m wondering if you can have a layer within the 
work you have, or maybe, I’m throwing a 10 million dollar research project at your hair, so 
I’m not realistically saying, do it now. But you go through this knowledge environment. You 
pick things up. And you say, “Well, I think this is bullshit, or I think this is important, or I 
think this relates to...”” Whatever it might be. But in a separate space, so that after a while, 
when they keep doing this, they get a better insight into their heads. And I can see Mark and 
Antoine making all kinds of head movements because I know it’s related to their work. But 
I’m wondering if you both have a brief comment and then we need to wind up. 

[Jamie Joyce]: I'll just say one thing is that, what you express reminds me of some exercises 
we do at the Society Library when we teach students about having them extricate what is 
meant and claims, it’s not the same thing in terms of defining things. But it is an exercise. 
And I will just say that the feedback we get from students, from performing logical 
deconstruction exercises pulling out all the claims and media, is that we’ve heard many times 
that, by the end of the semester they gain a new sight. Because they just inherently see the 
density of language in a way that they didn’t see it before. And it’s really lovely because so 
many of them get so excited, and they come back and volunteer for us. They’re really 
enthusiastic about what they’ve learned, and they find it to be very valuable. So I think that 
an exercise like that could also be very valuable just based on the feedback we get from our 
exercises, which are not the same, but similar. And then, what you expressed reminded me of 
is that, I just had the thought just now, of a new kind of exercise where students could be 


compelled to be presented an argument, and then counter-argue with it, and then they can also 
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see like how the Society Librarians actually steel man that argument. So they could work 
through the database only seeing partial pieces. And are doing their own research and 
counter-arguing. And then comparing that against the professional version. And maybe it’s 
better, maybe it’s not better. And that could help refine their thinking, as well. So, yeah. I 
think that those types of exercises really do help people with critical thinking. And I think just 
increasing their epistemic literacy, as well. Just really knowing how many assumptions that 
we pack in, to our everyday expressions, and understanding. And we’re forced to extricate 
that by defining or deconstructing. We really start to appreciate the density and complexity of 
meaning in knowledge. 

[Frode Hegland]: Fantastic. I mean, Tools for Thought is part of this big thing. Marc- 


Antoine, was that your hand or was that a little mouse? 


[Marc-Antoine Parent]: That was me. Sorry, I’m going to diverge a bit. And this is my 
thinking, not the Society Library. But the definitions, as you put it, is fundamental and what 
you’re doing, Frode, is helping encourage people to do their local contextual definition. And 
Jamie’s tool and work certainly does contribute to identifying specific definitions and work. 
And what I’m currently most interested in is these, how to assemble social definitions 
from individual definitions? And how to identify how they relate? Where are the 
differences? Including in emerging concept conversations. In some conversations, the 
concepts are emerging, the definitions are being negotiated, and evolving, and renegotiated. 
And this is where the ability to show the relationships between the concepts is extremely 
important. And when I say show the relationship, and that’s another thing I wanted to say to 
you, I think that the ability to qualify links is primordial. Bruno Latour, who’s a historian of 
science, has said that erasing the nature of links is one of the great crimes of 21st-century 
thinking. Saying these things are related. How? Doesn’t matter. That is absolutely terrible. 
Understanding how things are related is really key to a certain precision of thinking. And 
having a good epistemology and ontology of how things are related, I think is absolutely 
fundamental. And how things change when you push them from one domain to another, that’s 
Latour’s work. When you shift from one definition to another is when there are these shifts of 
meaning, which are necessary, they’re not always bad. Some of it is confusion, some of it is 
fluidity. But you need to be able to identify it. and that means naming the relationship. 
[Jamie Joyce]: Yep, I just want to quickly say because this may be relevant to everyone else, 
the conversation of definitions has come up in communities that Marc-Antoine and I have 
both been in. And you may all find this to be very interesting that the Society Library 
approach definitions are also descriptive. So, for example, when we actually create 


definitions, we can disambiguate them for the situation. Which is really important in the data 
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architecture. But also, we only give definitions to things that aren’t heavily contested. So the 
definition of climate change, for example, in the Society Library database, there are 19 
different definitions. That’s because it’s such a common phrase, that’s actually just a zip file 
of 19 different files. It’s the same name for 19 different zip files of meaning of what people 
apply when they’re using the term. So when it comes to modelling argumentation you can’t 
just say, “Okay, climate change”. Because so many people are just going to come to that and 
interpret different meanings. So, we’ve had to find different climate crisis, this is a little bit 
different than catastrophic climate change, this is a little bit different than global warming. 
All these different things. And then we’re just going to have to get to the point where we start 
creating new names. Like the climate change hoax. All these different small differentiations 
to let people know that we’re not talking about climate change in the same way at all. So for 
us that is actually a debate. Our primary question within the climate change database is: what 
has changed? And there’s 10 different arguments about what climate change actually means 
and entails and the evidence that support that it’s derived from different media sets, etc. But 
many of the definitions that are not contested like, what is the Diablo Canyon Nuclear Power 
Plant? Oh, it’s like a double loop Westinghouse, blah, blah, blah. That’s not really something 


people argue about, so we just give it its definition. 


[Frode Hegland]: And on that bombshell, thank you very much, Jamie, and everyone else. 
And we’re here every Monday and Friday. I look forward to having this transcribed, will take 
a while to organise and clean it up. And continue the dialogue and make amazing things 
happen. In this community, we’re looking forward to doing some sort of, a flatland, which is 
what we call, what we’re on now into VR environments. And back again a demonstration of 
the Future of Text at the end of the year. So it’s very interesting to have the insight, 
knowledge, and questions from today, and hopefully, some degree of dialogue collaboration. 
Have a good weekend everyone. 


Jamie Joyce: Thank you all so much for your time. Thanks for having me. 
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Jaron Lanier 


Keynote 


https://youtu.be/uZIO6GHpDd8 


I would like to discuss a few topics today that are related: 


1) I’m going to start with a discussion of whether the combination of computation with text, 
as we've known it, poses a danger of placing people into something of a trap, something of a 
loop in which we lose our future. That's topic number one. 

2) Topic number two is going to be whether text as we know it is something that we 
should think of as an eternal central feature of the human condition, or whether it 
might eventually become less important because other things come along and I will that 
will be topic number two. 

3) Topic number three will be about so much spiritual question of whether a degree of 
mystification of people or text is appropriate going forward in different ways than it has 


been in the past because of computation. 


So let me start with topic number one. In my peculiar life—and it is really a very strange life 
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that I can hardly believe—one of my roles is as what's called the prime scientist for one of the 
tech giants, which is Microsoft. And as it happens, our office is the funder and also the 
distribution channel for a lab called Open AI that you might have heard of that has created 
probably the highest performing so called AI text service now, which is called GPT. There are 
different versions of it. I'm sure you're all familiar with this. So it's essentially what our 
colleagues at Stanford have started to call a ‘foundational model’, a very, very, very large 
scale model in which all the available text that can be gathered has been gathered and then 


analyzed in the context of a very, very large computational system. 


The result is a service that can simulate a human interlocutor and actually perform in 
ways that impress people. I would say in general, for instance, it can often do reasonably well 
at passing high school or college level math classes, which even though there's no 
representation of math inside it, it's just one example. It can explain jokes, it can do do all 
sorts of things, reasonably effective translations between languages in some cases. And yet it 
also has this curious property of suddenly running into., very strange failures where it's 
obvious that it doesn't in fact have any representation internally of what's being talked about. 

So what do we have here? The first thing to say about this type of model? So what 
does this program actually do? Well, it really addresses the core feature of what we call 
text, which is sequence. Ultimately, text is a sequence of a number of things where the 
number of possible sequences is vastly larger than the number of original things that are 
sequenced. So the things that are sequenced are, well, in some languages the letters, but in all 
languages, the words. And if all we do is we capture the sequences and we can statistically 
predict what sequences are more likely than others, we can create a simulacra of language 
that is remarkably effective. 

Now, here on something interesting, which is that it's all about scale or all about the 
size of the model. 

If the model is too small, obviously it won't do anything, which is why this kind of 
illusion of a person inside the computer didn't really start to work until recently because we 
just couldn't build big enough computers. The ones that we build to accomplish this effect are 
truly vast. The they're the size of cities. They often are in remote places by rivers that can be 
used to cool them. They often have their own power sources, hopefully huge renewable, non- 
carbon emitting ones. There very, very few organizations in the world that can afford to have 
these models. Microsoft is one, Google is another. The Chinese cluster of companies closely 
related to the government are another. And there aren’t, there aren't really that many more. 

And the I would contrast this with a hypothetical but impossible infinite model. And 
this was imagined by Borges in the Infinite library. And of course, that one would be 


absolutely useless because it would take infinite time or energy to to get to any sequence in it. 
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And so it's effectively none. It's useless. Could it exist? Should it exist? And of course it 


cannot. 


So there's the the effect happens at a large scale, but not too large. And you might 
ask, do we know exactly what that scale is? Not really. We know the beginning of where it 
starts to work. We know it'll work even better at a larger scale. But there's some point at 


which it will start to collapse on itself. We don't know exactly where that peak is now. 
But what can we say about this? What have we learned about language? 


I think the first thing we can say is that we've learned that most language use is not 
creative. If we look at it from a global perspective, and we've never had a global perspective 
before._we’ ve never had a way of looking at everything everybody said—only locally at what 
somebody published and what somebody said in a conversation and so on. And the Internet 
doesn't give us everything everybody said, but it does give us everything everybody said on 
the Internet, which is a lot, especially for younger generations, and also for anything that's 
been in a library. It's also in the model. And the fact that regurgitating in a sense or 
interpolating between what has already been said can simulate somebody saying new things, 
tells us that from a global base on a global basis. There's a lot of redundancy or a lot of a lot 
of parallel restatement of things because that's the only basis for the solution to work, which 
perhaps shouldn't be a surprise. And yet it's different to actually have evidence of 
something rather than surmising it and now we have evidence for the first time of what 


language as a whole for everyone at once contains, which we didn't have before. 


Now what is of interest to me? One thing, one of many things of interest to me is that 
as we start to use these algorithms, not only just as a novelty to say, ‘Oh, isn't it cute that it 
can seem like a math student or a psychologist or whatever we might have at simulating?’ It 
is also of interest to me whether if we start to integrate these types of tools into our own 
conduct of life, whether we essentially increase the degree of redundancy and lose track 
of the possibility of creativity. 

Now, when I say this, I have to say I'm speaking in a way that strongly violates what we 
can call tech culture or the usual milieu that I function in, because there's a sort of a I would 
say a dogma or even an orthodoxy that there is not really such a thing as creativity, which 
sounds a little mystical, but instead there is some sort of a playing out of large scale 
recombination. And then eventually this turns into something we think of as creativity very 
much as Darwinian evolution is very creative, but in any particular instance, it's thought to be 
sort of random with feedback, and perhaps that's how everything is and that there is no 
creativity now. 

I think there's an important difference between the foundational models that we 


can build today and say Darwinian evolution for just one example. And that difference is 
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that Darwinian evolution is always about something. There can be some difference of opinion 
that persists to this day about exactly what it is about. There are some sort of hard nosed 
adaptation is to say it's only about survival. And then there are others. Including going back 
to Darwin, who would say, actually, there's sort of a some sort of aesthetic process involved 
sexual selection and other points of intervention where evolution seems to function a little 
outside the boundaries of pure survival and have an intrinsic creative quality to it. But at any 
rate. One can debate those things, but when it's very, very hard and I say this after having 
debated them for many years, it's very, very hard to come to a definitive conclusion on such a 
question. 

However, the when a program like a foundational program like the GPT generations is 
not really about a topic external to itself. It's regurgitating original conversations that in most 
cases were in the original instance about something. So it's it's a degree removed from being 
about anything, from being about a topic. 

And and so then the question is, is there a sense in which if we rely on these things, for 
instance, if this type of program is used as our tutor, there are many proposals that they 
become the universal tutors for kids or even adults in education. If it is used as a criminal 
detective, if it is used as a physician if it is used as well. Any other task were previously there 
might have been a human. 

Is there a danger that it limits what then happens to repeats of what has happened 
before? Now you'll find many who argue, especially with the tech industry, that we already 
are seeing creativity, and there are those who think that these things might be conscious 
inside or something. And once again, very much as with the question of adaptation to human 
evolution, it's extremely hard to get to an absolute definitive conclusion. However, it is not 
that hard to design situations to trip up the systems, which is not done that often 
because people actually want to see them succeed at simulating a human. There's a 
tremendously strong drive for that, which I'll get to in a second, which which I'm deeply 
suspicious of. 

So I was thinking about a sort of an irony here, which is the tech culture ethos or 
Silicon Valley ethos, if you like, is very much that the future will be not only different from 
the present or the past, but so different that it's incomprehensible. 

There's often talk of a singularity, which is when everything changes so much because 
of our advances in computation that we can't even recognize it, that everything transforms, 
that the whole universe becomes fluid in a new way because of nanotechnology spreading out 
to the edges of creation, always everywhere or something, something like that. And this 
happens all the time. All the time. There's a very frequent idea that these artificial intelligence 


systems will become so effective that they'll solve all of humanity's problems. You'll often 
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hear and I mean very often at like a dinner party that well, if you look at how effective GPT 
is, we can be assured that a program will solve our climate change problem. We can be 
assured it'll solve any problems related to infectious disease, it'll solve any problems related 
to supply of fresh water, etc., etc., etc.. So it's the only problem we have to work on and then 
that that will solve everything else now. So there's this. Let's leave aside for a second whether 
that hope is well placed or not or whether it's not, and I think it's not. But well, let's leave that 
aside. 

What I want to note is this sense of the one of the terms is exotropia (?? Jaron?). And 
there are many other popular terms within tech culture. This notion that the future is taking 
off and going into places that are unimaginable, that it's radically creative, radically 
different. Now, so when I talked about irony, what I mean is we built this thing that would 
appear to be regurgitated and profoundly nostalgic, profoundly trapped into 
interpolating things that have already been said. And yet that's in the service of this 
thing that's supposed to be profoundly future oriented. And I find that extremely 
interesting. 

It reminded me a little bit of the curious effect of Finnegans Wake, which I was 
rereading recently, where you have this this text that's maximally inventive with as many. 
Puns, puns, and weird ideas and double entendres and everything per word, as is conceivable 
in English, I suspect. In the service of depicting this being in this cycle in which nothing is 
really new, right and so there's this contrast between the nature of the text and what the text is 
about being almost opposite and I feel like we have a set of opposites like that in Silicon 
Valley now, but going in in the other direction where we have a regulative, fundamentally 
uncreative text depicting an infinitely creative future. So it's like the tech culture is the 
opposite of Finnegans Wake in a funny way now. This concerns me a great deal. 

The effect is even more apparent maybe not so much in text, but in visual art, where 
foundational models of visual stuff. So from the same lab we have something called DALL- 
E, and now there's some other versions of it, like stable diffusion. Some of you might have 
seen where you can ask for a piece of art where you can say, ‘I would like to have rats flying 
in a flying saucer in the style of Turner’, and it'll produce this thing and you think, Wow, this 
things are creative. And yeah, and yet it is fundamentally regurgitated. It can eat, it can input 
the style of Turner and regurgitate it, but it cannot be a Turner. And also another interesting 
thing I mentioned failure modes. It's easier to see them in the visual. 

If you look at the images that are produced by the visual foundational models like 
DALL-E, they can be very impressive doing such things as what I just made up. And I mean, 
not that I've tried that particular one. You never know for sure, but probably if you ask for 


rats in a flying saucer in the style of Turner, it would do it and it would probably looked at, I 
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don't know. Somebody can try it now if they have it open. But if you ask it to do hands or any 
creature with hands still tend to not be good, the fingers will be mangled. And the reason 
why is that hands have structure, they’re not just surface, the hand has to make sense as a 
functional hand, and interpolating between images of hands tends to mess up that thing. So 
you tend to have a lot of mangled and weird hands or hands that don't quite work now as is, if 
you point this out within those who really want to believe in these things as being alive, you'll 
find the excuses are that people often can't draw hands, which is true. So there might very 
well be some degree of similarity between what goes on in a human brain and what goes on 
in these programs. I would say that it is beyond our current horizon of science to say how 
much similarity, but it does seem reasonable to say that there's a little bit or some amount. I 
don't think it's total, but I think it's zero. 


I think we overstate the similarity when we call the accumulators and set our models 
neurons. And I think the term artificial intelligence (AI) overstates the similarity, and yet 
there might be something there, I have to say. 

Now, this problem I was bringing up of a regurgitate of culture filtered through devices 
that we can build based on recordings of our past behaviors and communications 
reminds me a little bit of how many systems that are related to something in the world that 
might involve representation of it or response to it can become overly narcissistic, if you like, 
or self self self oriented. So for instance, the immune system can generate autoimmune 
diseases and economy can become focused on artifacts of itself and become dysfunctional, 
leading to market failures. Many, many other examples. And so there's there's a if we can 
think of these things instead of as people as a representation system that's vulnerable to... 
(system announcement by LiSA: It’s 4 PM) It is 4 p.m.... You can think of this as a system 
that's vulnerable to becoming trapped in a self reflection rather than being responsive 
to the thing that it's supposed to be aligned with. And that would be a maybe a less 
charged way of stating the concern I have about regurgitated culture, but ultimately I want to 
get to sort of a mystical level of this. 

And what I mean by mystical is in a way kind of literal, if mystifying, an aspect of 
what we're doing instead of attempting not to. So, I have long held. And when I say long, I 
mean my one of my mentors when I was quite young was Marvin Minsky, who was probably 
the most important source of the current ideas about artificial intelligence in terms of the 
images and cultural references that are in use. 

I used to argue with Marvin when I was young, and Marvin, having been one of the 
original generation of people who believed in artificial intelligence (AI), loved the argument 
(but people, as is always the case, the subsequent generations become more orthodox and 


lose their sense of humor and don't have the kind of charm and openness that the original 
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people had, at least in person). So it's my belief that you cannot have perfect ideas. I don't 
think there's such a thing as a perfectly completed science or perfectly completed 
cultural theory or perfectly completed mathematics. Any time we apply thought or any 
time we conceive our world or ourselves—we do so with fractures. That doesn't scare me. I 
think it's a miracle that we can do anything. The fact that we can even do it partially or 
imperfectly is fantastic. For some it bothers them that they can't achieve perfection. I don't 
know why that should bother them. 


Here's what I want to propose in the future that we must think about where we 
place our fractures. And we do have through some miracle, which is the beginning of my 
mystification. Through some miracle, we have the ability to choose to degree where the 
fractures will be. And what I would propose is that instead of trying to say, well, here we 
understand what an intelligence is and we can reproduce it in a machine which then offsets 
the fracture somewhere else, because then you have the problem of trying to explain, well, 
where did all this language come in the first place? That's been input into the machine. You 
end up pushing the fracture out backwards back in time or to some other spot. 

I think the better thing is to put the fracture inside the person. In European thought we 
went through a process of recognizing that you can't prove the existence of God. It's a matter 
of faith. I think the new thing in response to these models is that we have to start to have 
a radical, mystical belief in the existence of people as the source material from which 
these models can be built. And we have to treat ourselves as mystical, transcendent sources, 
as sort of supernatural, because any other alternative puts us into a regurgitated trap and 
puts and makes us subservient to creations that will become self resonant and limit our 
world, and also concentrate wealth and power unsustainably among the nerds who run 


the models. 


I only got to one of the three things I was going to talk about, but there you go. 


O&A 


Frode Hegland: I'd love to hear a little bit more about what you mean by fracture. Thank 
you. 

Jaron Lanier: By fracture, I mean the limits to the ability to make a consistent and complete 
and perfect assessment of oneself or one's world. In mathematics, we have many such 
fracture fractures, the most famous maybe being girdles theorem, but this is also true 
everywhere. If you look in in the sciences, in physics, you can push, push, push, push back 


to, I don't know, the big Bang or something, but there's always some kind of an artifice that 


361 


you have to make up at the edges of what you can understand and you can push it back, but 
not infinitely. And that's formal. I mean, there's just no there's never going to be an absolutely 
complete, fully rounded circle, closed physics. It can get better and better, but not perfect. 
Similarly, I think our understanding of text. It can become deeper and deeper. But I think 
ultimately, as I say, there's always going to be a bit of a mystery about how this thing works 
at all. We can offset that mystery to something other than text, but we can't offset entirely 
because we can't make a completely consistent universal view. So I'm proposing that we 
position the fracture and the person in order to mystify the person and make us sort of special 


and supernatural rather than machines or any other artifice. 


Fabien Benetou: Thank you. So I'm wondering if one way to put it, is that the trap? I mean, 
is it a genuine trap in the sense that we all lose agency or that the loss of agency is just 
temporary, or is it just for some of us or maybe we actually properly earn agency? 
Jaron Lanier: Yeah, well, you know, this was a theoretical question 20 years ago that I used 
to talk about and write about quite a lot and. If anybody is interested in looking at my early 


concerns about it. There are some essays from the nineties. One is called ‘Half the 


Manifesto? ’, I think, and the other one's called Agents of Alienation2®. So that was when it 
was purely theoretical. But at this point it's done, it's empirical. 

So the, the text management programs that we call artificial intelligence are 
overwhelmingly used for the manipulation of humans now and not for any discernible 
productive purpose. And this is, of course, the problem with the social media companies and 
what we've seen. The answer is not 100% one thing or another thing. It's a statistical 
distribution. It's definitely the case that these things have reduced agency among people. 
For instance, they've reduced rationality and increased mental disease, I would say in people 
as a whole, and this has been studied very widely. They've decreased the quality of political 
discourse very widely, and this is also been studied. And so you see a statistical degradation 
of what I would call sensible autonomy in people when they are exposed to the algorithms 
thus far empirically. Now, as with, it's important to understand that. There might be a way of 
in fact, I think there probably is a way of incorporating these algorithms into life that doesn't 
have this effect. And it's not the algorithms per se, it's the algorithms combined with an 
economic incentive because of stupid business models, the so-called this is the whole thing. 
Anybody who wants to read this, I've written about it a lot, obviously so. And it's also stupid 
philosophy. So it's bad economics. Bad philosophy combined with the algorithms that make 
the algorithms destructive of human autonomy, if you like, and dignity. I think the algorithms 
actually can be useful, and there's no reason for them not to be. But it requires a change in 


philosophy and economics. 
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Max Drake: Hi. Max Drake here. I really like what you were saying about mysticism. And I 
think as someone who's worked a lot with GPT three so far and seen Internet responses to it, I 
can definitely imagine a kind of like a dueling. I just the bottle itself, kind of. Oh, sorry to cut 
out it. Kind of. Yeah, so I can imagine. Yeah, there's like the possibility of some other kind of 
mysticism or merging that is more nefarious in places, the model itself as kind of the source 
of that. And I was wondering, is that what you see as a kind of like is that similar to. Is that 
what we need to fight against or is what's your imaginary for? 

Jaron Lanier: You know, right now people roughly speaking, there's two ways of interacting 
with big models online. One of them is where the model is kind of intrinsic to the interaction 
and the other one where is where it's explicit and you know about it. So, so far with releases 
like GPT people, no, they're interacting with GPT, that's the whole point. And, and, and so if 
the person is led in on the joke, so to speak, if there's an awareness of what's going on, then I 
think then what you were calling the nefarious nature of it is greatly reduced. And so and in 
fact it's in those terms. So one of the problems that a lot of technical culture is formed when 
people are, especially people with any technical skill or interest, are interacting with these 
things. And in that case, they're cute. I mean, it's not nefarious, but then when the transition 
happens is when people are using a social media system or anything that has 
recommendations or anything that constructs an experience feed. And instead of being said, 
well, here's the model, here's how you can tweak it, here's how you can play with it. It's just 
intrinsic to their experience. And in that case, it becomes sneaky and very subject, in fact 
explicitly subject to corruption, because the whole point is that third, third parties are paying 
in an attempt to control the attention and manipulate the people who are using it. And that's 
all the Facebook or media companies do that. TikTok does it as an example now, and they're 
all they're all doing that. And that is that is where the damage comes in. It's when people are 
having experience created by algorithms that you start to see degradation of human decency 


and and intelligence. And it happens so far universally. 


Andreea Ion Cojocaru: I would be interested to know where you personally, what you 
personally see as the main source of mysticism or potential mysticism inside people. Are 
you are you primarily looking at consciousness or some of those things like you personally? 
Jaron Lanier: Sure. Well. There has been a strong wind of sort of anti consciousness 
acknowledgement in technical culture for decades and. This. I, I disagree with with that 
tendency. The the argument is something like we used to think the earth was the center of the 
universe, and now we should recognize that our consciousness is at the center of anything 


and we're not special and all that. And at first it seems kind of humble and kind of in the line 
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of the of the Enlightenment. But I think actually all it does is it forces the mysticism 
somewhere else. And for instance, people who talk about AI algorithms and believe in that 
they're conscious or something will never stop sprinkling magic dust on the algorithms. Oh, 
it's so magical. It's come alive. But then they'll say for people, Oh no, these people are just, 
you know, it's a kind of an inconsistent thing. Like what you're doing is you're you're moving 
the magic dust from people to the machine. But the machine is owned by some company. 
And it's it's it's politically and economically a terrible idea, but it's also just philosophically 
sloppy. And I just think we should admit that we can't get rid of magic dust and might as well 
put it on the person. There are many, many angles on this, and I've been in the consciousness 
arguments for many years, but. I would say consciousness is the one thing that cannot be 
reduced if it's an illusion. And we should treat it as a uniquely efficient place to put our 


themes. 


You know, it's like the most sensible place. And so consciousness is some kind of 
impure. It's some kind of a channel that's not empirical, it's something else. I don't think you 
can prove it. I used to sometimes argue with people who were skeptical of it, like Daniel 
Dennett, that the only way you can. There are some people who are professional philosophers 
who claim not to be conscious, and maybe you can believe them. But in general, one should 
have the faith that other people are conscious, and maybe that's the more appropriate and 
useful contemporary faith instead of faith in God. It's similar, actually, but it's slightly 
different. It's just. And then the other question I would ask is, if we're going to create a 
society that's run by algorithms, if we don't elevate people in some mystical or supernatural 
way, how can it serve people if it's all just information components, why doesn't it just 
devolve to whoever owns the computer and serving them? I don't see any other way unless 
you make people special and you see that in the early Enlightenment documents about 
democracy and society, You see this recognition that you have to just treat people special and 
there's no ultimate logical justification. There's a there's a pragmatic reason, given a set of 
opposing beliefs that can't be resolved through logical competition, that you have to become 
pragmatic. And weirdly, we have come to a point where mysticism is the most pragmatic 


choice, as well as, I believe the most philosophically gracious one. 


Vint Cerf: Hi, Jaron. It's so good to see you. Thank you so much. So it's been I'm sorry you 
couldn't be here in person. Actually, this isn't the question. It's an observation, as usual, 
listening to you as an intellectual romp and several new phrases have occurred to me as I 
listen. The first one is stochastic retrieval, which is basically what a lot of these things do. 
Second, casual retrieval, which is what happens when you have the dialogue. And I really 


like your idea of the universal computer. These things know more than we do, although they 
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also know false things as well as true things. And that's a problem we have to deal with. And 
the last point is that since this is really recreating the already existing dialogue, so to speak, 
human discourse, this could be used to create a dialogue with a dead person. And that 
suggests that another label for these is rhetorical zombie. 


Jaron Lanier: Let us be clear about the power relationship here. And so nice to see you. I 
what? This tendency to want to revive the dead using algorithms is very, very widespread in 
tech culture. And I. I think. We should treat it as an evil. I understand there could be some 
circumstances where it could have utility and there are often scenarios discussed where, oh, I 
don't know, some kid has a traumatic loss of a parent. I did when I was when I was young, by 
the way. I lost my mother when I was young. And then this notion that maybe it could be 
therapeutic. I think we should. Adopt. It's kind of interesting looking at the Islamic resistance 
to representing people as images that that has been traditional. Perhaps I'm not proposing that 
we adopt that. And yet I think looking at the impulse in its source is instructive and 
worthwhile. I, I think there's a terrible danger in telling a kid that the parent can be 
represented in code. I think demystifying the parent, turning the parent into something that's a 
portion of a database or an algorithm demystifying the parent will inevitably instead mystify 
the computer or whoever provides the service or something. There's no way to remove 
mystification because there's no way to have a complete point of view. So the inevitable 
conclusion of simulating a dead person is to subsume that person into somebody else's 
scheme. Just given how politics and economics work. And so I think we should treat it as an 
evil and I would like to see it become treated as a moral outrage, possibly even illegal in 
some circumstances. I really do feel that this is a road to civilizational ruin. Few agree with 


me, but I think more will come to see the merit in this concern. 
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Jim Strahorn 


The Future of ... More Readable Books ... a Reader Point of View 


One should NOT have to read an entire article or chapter to understand what it’s about. 


Unfortunately, I read slowly. I remember far less of what I’ve read, than I would like. 
I highlight to understand. That slows me down. I try to scan, read and skip selectively. 


Like most people, I'm a little smart, not brilliant. I can't read and retain entire pages. 


Many writers waste reader time in not communicating their main points more effectively. 


I'm talking about books focused on specific topics, problems or opportunities, things that 


affect our daily lives, technical books and subject-specific books that draw conclusions. 


Textbooks seem a lone exception: they typically are heavily formatted for reader benefit. 


I'm NOT talking about fiction, the great novel or narrative stories that flow across time. 


I'm arguing that many books would benefit from being more like textbooks than fiction. 
The Problem: 


English teachers and schools teach style, great literature and writing, of fiction not fact. 


They don't teach organization, structure, content hierarchy, sub-titles and formatting! 


Most books thus have no subtitles, no bold text and minimal formatting. 


They have endless paragraphs of unformatted, unsub-titled oceans of text that readers have to 


search for and struggle with to find the author's main message and conclusions. 


Some authors seem to write technical or topic-specific books as if they were writing fiction, 


as if style, flow and exemplar stories are more important than message clarity. 


That's makes reading, absorbing and understanding an author's message very difficult. 
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Objectives: 


Make non-fiction written communication more effective for the reader. 
Authors should be making what they're trying to say to readers more explicitly clear. 


They typically don't use the simple tools available to them to help readers understand. 


Make section headings, sub-titles and bold-text more the typical norm. 


The general focus of a document, it’s conclusions and major messages should be obvious to a 


reader, visible at a glance, NOT buried in oceans of run-on text all the same tiny size. 


Specific Format Suggestions: 


Structure and format text to facilitate quick scan and selective reading. 
Most of us have too little time, and careful reading takes time, regardless of profession. 


On occasion, when frustrated by what I was reading and by its near total lack of sub-titles and 


formatting, I've taken time to reorganize and reformat other people's written work. 
I've been surprised by how much more effective the reformatting experiments have been. 


Rather than trying to convince anyone here that a "structured format approach" is preferable, 


I'll simply suggest that authors and readers do their own short experiments. 


Make important key ideas large and bold ... so visible at a glance! 

So what is important looks important:! So major message can quickly catch one’s eye. 
So key ideas are self-evident and captured in short, single line sentences or short phrases. 
A reader's eye naturally jumps from one Bold statement to the next Bold in a sequence, 


skipping the lines of text in between almost automatically, as if they weren't even there. 


Typically, important ideas get lost in the ocean of identical text. That seems a great way to obfuscate an author’s intended message. 
Books are writer-centric, written, and carefully edited by colleagues A 250-page book has, say, 30 lines/page and 10 words per line, so 
and editors for grammar, spelling and clarity of author message. 300 + or - words per page, so 75,000 total words in the book, that 


But a typical 20-50 page chapter has no Chapter Table of Contents, capture the big ideas, support logic, related detail, background 
no sub-section breakdowns, no sub-titles, no bold text, no type size information, context, citations, and some transitional language. 
difference, and no formatting, highlights or emphasis of any kind. —_Yet those 75,000 total words are all the same identical tiny size! 


Example 1. Strahorn, 2022. 
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Prioritize content graphically, in layers of importance and declining size 
Authors should give readers a sense of the structure and relative importance of content. 


Content typically contains hierarchies of information, that range in declining order from 


specific major conclusions, component points, related logic, reasoning and support detail. 


Such hierarchic organization, unfortunately, is difficult in narrative story-telling mode. 


Imagine driving cross-country in an unknown foreign country Use few, if any, sub-titles, section-headings or bold formatting? 


without roadmaps, road signs, numbered highways, street names Use many more words on transitions between chapters and/or sub- 
or knowledgeable strangers whom one can ask for local directions. sections, when sub-titles or chapter structure makes transitions much 
That’s what most authors and editors seem to do to their readers! more natural and visually obvious, both to the author and reader. 


Example 2. Strahorn, 2022. 


Short sentences and paragraphs generally are preferable to long. 

Keep sentences to a single line, if possible, and only two or three lines if they're needed. 
Sentences are easier to read it they start at the left edge of page, not mid-line. 

They're easier to understand because key ideas are more visible, not lost in text. 


A reader's eyes and mind get lost all too often in overly long run-on paragraphs. 


Key Ideas in Short Phrases remain in mind more than long sentences. 
Short-phrase subtitles enhance a reader's recall of the author's content and related logic. 
They facilitate reader retention and recall; they trigger related associations in our mind. 


In fact, it's those associations that makes the key-idea-short-phrases effective sub-titles. 


Use Section Headings and Sub-Titles to ... 
- help readers scan, skip and read selectively ... and more purposefully 
- make document structure more visible to the reader 
- keep the reader better oriented within the author's content and flow 
- divide pages of endless text into more digestible chunks 
- help the reader listen and stay focused on what the author is saying 
- provide short, keyword phrase summaries that are easier to remember: 
a visual image of a sub-title is more memorable than 2-20 lines of text. 
- trigger associations with a reader's existing knowledge more effectively 
- make content easier to read, absorb, review, re-read and understand 


- utilize the ways our human minds work more effectively 
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- make remembering content easier in the short-term 
- facilitate improved reader retention in the long-term 
- Sub-titles are road signs for reader use, guides to the path ahead and behind. 


- Thus, they're also a shorthand overview or summary of a document's structure. 


Effective formats clarify both author message and reader understanding. 

Our eyes can capture paragraph essence at a glance if the page effectively formatted. 
Text formatting, done right, enhances both reading speed and retension of content! 

The texture and appearance of words on a page affect ease of reading in beneficial ways. 


Graphically prioritized formatting clarifies document structure and information priorities. 


Few books have adopted PowerPoint's focus on just a few major points. 
An author’s most important ideas and statements are too often buried in volumes of text. 


Supporting details are far less relevant if the key ideas are not visually self-evident. 


Few books have replicated the communication impact of a good video. 
Why is a 20-minute video interview often more insightful than a 200 page book? 


Because the author is summarizing the book’s major points, not the details and the reader’s 


time is typically too limited to invest 3-6 hours or more reading that book. 
Written text, ideally should be structured, written and formatted to allow one to 


scan, skip and read the book selectively, much-like a 20-minute video summary. 


Conversation often uses incomplete sentences; so should our written text! 
The missing parts of the sentence are implied. And generally understood by both parties. 


Formatting written text is analogous to modulating one’s voice in a conversation, where 


changes in pace, volume, a pause or the use of hand gestures for emphasis are the norm. 


We can use Ellipses ... for pauses and CAPS for emphasis ... as in conversation. 


Lists buried in paragraphs are more effective if formatted as lists. 


When an author announces three or four points or conclusions to follow, that three-ness or 


four-ness should be visually obvious to the reader, and a list makes that very clear. 


Readers should NOT have to waste time searching paragraph or next pages to find the next 


item in a sequence when it should be visually obvious and easy to find at a glance. 
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List formats make reading and review far easier, as in the two paragraph examples below: 


EXAMPLE: Traditional Continuous Text Paragraph 


The books I read are typically about politics, democracy, the dysfunctional political system in 
the United States, money in politics, the dominance of corporate power, and the rigged 
political system, capitalism, giant corporate capitalism versus decentralized entrepreneurial 
capitalism, capitalism's strengths and weaknesses, rising and income and wealth inequality in 
the U.S., economics, investments, real estate, the stock market, the Federal Reserve Bank and 
big banks in general, the causes of the S&L crisis and the 2008 financial crisis, intelligence, 
the human brain, body and mind, psychology multiple personalities, collective intelligence, 
team chemistry, knowledge maps and dynamic knowledge repositories, computers, software, 
neural networks, society of mind, and computer aided design, architecture, urban design, 


design thinking, and design, in general. 


EXAMPLE: Same Paragraph Formatted as List with Seven Subject Groups 


The books I read are typically about ... 

- politics, democracy, the dysfunctional political system in the United States, money in 
politics, 

- the dominance of corporate power and the rigged political system 

- capitalism, giant corporate capitalism versus decentralized entrepreneurial capitalism, 

- capitalism's strengths and weaknesses, and income and wealth inequality in the U.S. 

- economics, investments, real estate, the stock market, the Federal Reserve Bank, 

- big banks in general, the causes of the S&L crisis and the 2008 financial crisis 

- intelligence, the human brain, body and mind, psychology and multiple personalities 


- collective intelligence, team chemistry, knowledge maps and dynamic knowledge 


repositories 
- computers, software, neural networks, society of mind, and computer aided design 


- architecture, urban design, design thinking and design, in general 
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Conclusions: 


I want to acknowledge that these Format Suggestions & Conclusions are my opinions, 


based simply on experience and common sense, and not based on any scientific research. 


Authors are in charge here, so they must make a conscious choice to: 
- take the traditional, quick, less effective, no formatting approach, or 
- format more extensively with the reader in mind, 

clarify their own thinking and understanding, 

communicate more effectively and deeply, 

enhance their own message significantly, and 


enhance the reader's understanding far more deeply. 
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Kalev Leetaru 


Seeing Through Others’ Eyes: Reimagining How We Experience The 


News 


Two decades ago, I built a CAVE virtual reality application called ShadowLight. Users 
stepped into the 10x10x10’ cube of the CAVE, donned their VR glasses and picked up a 
6DOF tracked game controller that allowed them to quite literally “draw in space” all around 
them. ShadowLight enabled both organic (freeform artistic drawing) and architectural 
(constrained CAD-like geometry) creation by reaching out in space and physically drawing 
with one’s hands, creating entire worlds out of thin air. The ability to create in physical space, 
yet selectively disregard gravity, fundamentally altered the creation process. Rather than 
create structures from the bottom up, designers suddenly created at will, creating forms and 
spaces at random in space and then forming the rest of the world around them. 

Uniquely, ShadowLight allowed the embedding of complete dynamic miniature worlds 
within that creative space, from physics-governed objects to connections to the outside world 
where objects and entire subworlds could be remote controlled or evolve based on the real 
world. Traditional desktop-designed CAD and artistic objects could be added, alongside 
realtime capture data from the real world, blending and bending the very concept of “reality.” 
The real world could be brought inside in realtime. Historical events could be replayed. 
Simulations of real and fantasy worlds and objects could play out. 

All of this played out in the same shared virtual space that a designer could create 
within, finding inspiration in ways never before possible. This single application was used by 
architects, designers, artists, engineers, planners and even middle school students, 
necessitating an interface that could be used intuitively by all. I personally spent two entire 
weeks living exclusively within this virtual world non-stop as an experiential experiment. 
One of the greatest lessons it taught was the power that comes from connecting the digital 
and physical worlds and using them to see the world through others’ eyes. 

What does it mean to experience and preserve the world to allow others to see a place, 
event or moment in time through one’s eyes? My undergraduate thesis involved capturing 
more than a quarter-million photographs of the University of Illinois campus over the 
seasons, including its major events, and digitizing tens of thousands of pages of books, 
pamphlets, letters, maps, proceedings and documents spanning more than a century and a 


half, the majority of which had never before been available online, together with writing the 
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histories of more than 300 buildings and spaces and hundreds upon hundreds of objects and 
events, weaving all of it into a massive digital experience that tells the story of the 
university’s evolution over the years. It remains, twenty years later, the definitive digital 
guide to the institution’s physical history and source materials. 

Most importantly, this history tells the story not of the lives of famous people, but of the 
buildings and spaces that define the experience of those who have attended or worked at the 
institution over the past century and a half. In other words, it inverts the idea of how we tell 
the story of our institutions. The average university student can’t rattle off the names of the 
deans and department heads of their college, but they know the buildings they pass by and 
through each day. By telling the story of the university’s history through its buildings and 
spaces, the narratives that emerge connect with today’s students in a way that traditional 
histories cannot. 

One again, traditional concepts are reimagined in a way only the digital world makes 
possible. 

Following in this tradition, a decade ago I pondered the question of just what defines 
the concept of a “book.” 

In our earliest years of life, books are visual-first mediums, filled with pictures and few 
words. In short order, however, pictures give way to words as we progress through school and 
our very concept of what defines a “book” becomes built upon words in place of images. 
Indeed, as the world of libraries has entered the digital era, book digitization has focused for 
the past half-century on capturing the text on the pages and discarding the imagery that 
appears alongside. Even the physical scanning processes used to digitize books have often 
used imaging sensors, lighting, processing and storage workflows optimized for text, at the 
degradation of images, to the point that bitonal black and white scans dominate many 
collections. 

What would happen if we returned to the books of our childhoods and reimagined books as 
collections of images, rather than of words? Of libraries not merely as archives of knowledge, 
but as the greatest distributed art gallery ever created? What untold treasures of artistic 
creation and historical record lay in wait in the world’s books to be discovered? The end 
result involved extracting 12 million images from over 600 million digitized public domain 
book pages dating back 500 years from over 1,000 libraries worldwide and making them all 
browsable and even searchable by connecting each image with its caption or surrounding 
text, book metadata and other images within the same work. While the underlying books had 
been available online for years, these 12 million images had been scattered and buried deeply 
within their half-billion pages, secluded and invisible among their hundreds of billions of 


words. By extracting them from that text, mobilizing them andplacing them front-and-center 
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in their own right, half a millennia of images were now accessible and discoverable on their 


own terms, reintroducing them to a new era of audiences. 


Once again, the digital world afforded the power to reimagine the most mundane 


aspects of our informational lives to transform them into something extraordinary. 


Yet even this endeavor still treated books as closed end-to-end narratives written once 
and read unaltered for eternity. Could a book be more? 

What precisely is a “book?” Is a Choose Your Own Adventure a book? Is Wikipedia a 
community-contributed live-edited book? Is social media a book? Is a game or virtual reality 
experience a book? What if we stopped thinking of “books” in terms of physical objects 
printed once and distributed unaltered and began to think in the broader terms of the 
collection and compilation of information? 

The digital world is in essence a globally distributed live-edited ephemeral compilation of 
information, narratives, beliefs and emotions encompassing the planet — a form of book. That 
book’s authors represent just a fraction of the world’s communities, narratives and languages 
and its pages are unevenly ephemeral, with some lasting decades while others perish within 
moments of being written. Within its pages lay traces of the world beyond its reach. Unlike 
the two-dimensional world of the printed book, this digital world spans every modality, from 
video, audio, still imagery and text to the interactive world of code and the experiential 


virtual worlds they enable. 


What if we thought of the world’s news media as a form of live-edited live-streaming 
“book” encompassing global narratives and events? How might we make this live archive of 


human society accessible for scholarly and journalistic understanding? 


The informational world is filled with scholars writing and lecturing, but all those ideas 
are for naught if they never become reality. Rather than talk about how the world should be, 


today’s digital world gives us the power to create those visions, to bring them to reality. 


It is not enough to merely archive and preserve the digital world. It isn’t helpful to say 
years later that buried within petabytes of data in a digital archive were the earliest warning 
signs of an impending pandemic or the answer to the world’s most pressing questions. For 
such archives to be useful, they must combine human and machine interfaces to transform 
petabytes of data into actionable insights and understandings of the world. Can the sentiment 
of news predict wars, its undercurrents yield the earliest glimmers of tomorrow’s pandemics, 
its cycles forecast the future to come? 

Importantly, news is multimodal, spanning video, still imagery, the spoken word and 
text in all the world’s languages. Historically each of these modalities and languages were 
treated individually, but the narratives that govern our understanding of the world around us 


do not. All of these modalities and representations interplay organically in the form of 
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societal narratives and inorganically in the form of influence and amplified falsehoods. 


Al is increasingly used to process the firehoses of content that both define the digital 
world and defy human attempts to make sense of it. But, what does it mean for AI to “see” 
the world and transcribe it into codified form? This spans not merely the ontological lenses 
such approaches enforce, but the very question of quantifying a fundamentally qualitative 
world, stripping away representation, framing and ascetics to codify. What does it mean to 
distill an image of government security forces using violence to suppress their own citizenry 
into a machine summary of “police, protesters, violence”? As we explore how best to teach 
machines how to “see” a world made of news, we learn too how to help make that world 


more accessible to those with different abilities in the visual-first realm of VR. 


How do we teach journalists and scholars how to use these new analytic lenses? For 
more than six decades we have taught researchers and, in turn, society itself, how to think in 
terms of the humble keyword. From their earliest years, children the world over living in 
digitized societies learn to condense their most complex and nuanced questions underpinning 
every corner of their lives into a sequence of simplistic keywords to be typed into a search 
engine. 

What will it take to teach future generations how to use the post-keyword world of 
search? 

What does it mean to “visually” search television news? What are the dimensions that 
best define how we see the news and the information it conveys? How do the universe of 
objects, activities, landmarks, onscreen text, colors, shapes, textures, visual relationships 
come together to form an understandable visual narrative? How do we make all of that 
searchable in a way that preserves not merely the binary existence of individual labels, but 
the complex and interdependent relationships amongst them? Most importantly, how do we 
teach researchers and society at large how to think in terms of the fundamentally new 
metaphors and mindsets required to engage with these new richly condensed representations 
that codify that which is inherently qualitative and whose representation depends on so 


heavily on its realtime interpretation through the lived experience and context of the viewer? 


The global open data GDELT Project represents precisely such a firehose over planet 
earth. What does it mean to “see” the world as a firehose of events, narratives and emotions 
spanning video, imagery, audio and text in all the world’s written, spoken and visual 
languages? If the coming “metaverse” is to be a unification of infinite discrete virtual worlds, 
GDELT represents the “metaverse” of the global journalism landscape, constructing a 
singular unified realtime digital image of planet earth that bridges our fractured and divided 
societies into a single “news metaverse.” The challenges of constructing a singular 


representation that spans the infinite richness of our diverse world resists singular definitions, 
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necessitating the construction of infinite immense intrinsically interconnected graphs that 
represent the world in all its infinite localized representations and connecting them, allowing 
inquiry that spans this representational continuum of preciseness to unification. What 


constitutes a “feast” or “happiness” across the world? 


VR is a visually-centered medium that has been historically dominated by Western 
visual storytelling. Yet, the web and social media have their roots in textual representation 
that is increasingly globalized. GDELT’s collaboration with the Internet Archive’s Television 
News Archive spanning 50 countries over 20 years captures the unprecedented possibilities to 
understand how visual representation and narrative traditions across the world have adapted 
to the shared medium of television news. All across the world, the medium’s constraints are 
the same, so how have the societies of the world adapted their storytelling traditions to it? 
These insights can tell us much about how societies will adapt their different visual languages 
into VR. 


Television news across the world is both highly similar and incredibly different. Some 
channels tell the news primarily through newsreaders in studios, others rely more heavily on 
first person reporting from the field and still others tell the news largely through political and 
religious leaders, panel discussions and interviews with ordinary citizens. Some channels 
focus on hard news, others on commentary and some use news primarily to promote 
government and cultural narratives. Some countries rely on commercial advertising to 
support television news, while others focus more heavily on music, cultural programming, 
public service announcements, government statements or religious content between 
segments. Studio environs range from staid to ornate to technicolor vibrance. Even 
presentation styles vary from speakers primarily facing the camera to primarily looking 
offscreen. All of these differences and similarities will once more play out in the coming 
metaverse, with television news having much to teach us as to the forms the metaverse will 
take. 


What are some of the challenges our metaverse future will hold? 


Globalization 


We tend to live within the confines of our own geographic, linguistic and cultural affinities. 
In the digital world, machine translation and global accessibility make it possible to achieve a 
transcendent consciousness that allows us to absorb the world around us. When GDELT first 
launched its Translingual initiative 8 years ago, there was widespread rejection within the 
social sciences of the need to look beyond English and the Western world to understand the 


rich diversity of the world’s societies. The idea that Western media was merely an 
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internalized and highly biased lens that shaped awareness and understanding of the world was 
largely rejected through much of the social sciences in favor of the idea that the informational 
landscape was abjectly neutral and captured all “worth” reporting upon, without an 
understanding of the values and biases encoded in that concept of “worth.” As the world has 
become ever more globalized, there is growing recognition of the need for a more holistic 


and globalized understanding of the world. 


From Firehose To Awareness 


How does one take this realtime digital proxy of the planet and use it to understand the 
conflicting chaotic cacophony that is our global world? The combination of machines sifting 
through trillions of datapoints in realtime to identify the anomalies that are the earliest 
glimmers of tomorrow’s biggest stories and working with humans to contextualize and 
understand their importance and that actions that must be taken in response represents 


fundamental new challenges in both technology and mindsets. 


Falsehoods 


Much like its physical predecessors, the digital world is filled with falsehoods. Its global 
reach and cheapness of distribution has dramatically increased the ability of falsehoods to 
span the globe, while the global competition of the information space means even the world’s 
mainstream traditional media finds itself in a race towards negativity, sensationalism and 
clickbait as its distribution is increasingly controlled by social media gatekeepers. The ability 
of such falsehoods to transcend the digital sphere to wreak real-world havoc and violence is 
reinforced each day. Yet in a digital world that spans the world’s rich diversity of societies, 
who defines what is “truth” and “acceptable” in a globalized world with very different 


understandings? How will that work in a future metaverse? 


Our Ever-Evolving Language 


All language models represent snapshots in time, yet the world is fluid. Sentiment 
dictionaries capture “cool” as “cold” or “extra” as positive. Words, images, even emojis are 
constantly redefined and taken ownership of by new generations and communities, their 
meanings fluid and ever-changing. Machine translation models today still fail to 


properlytranslate pandemic-related terms that did not exist when their massive models were 
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trained and their size, scale and brittleness makes them difficult to update. The digital world 
is perhaps the world’s largest global observatory for changing written and visual language 
throughout the world. We can see the emergence, decline, evolution and transcendence of 
language in all its forms evolving across time and space. Much as the web today encapsulates 
the detritus of decades of society, what will the VR world look like as the ever-changing 
landscape of our visual narratives and metaphors must suddenly be accounted for, building in 


aggregate and uneven ephemeral layers over years? 


Preservation 


What does it mean to “preserve” the world’s news media? The balkanization, geotargeting, 
personalization, ephemeral perpetual editing and recontextualization of the news means it is 
not sufficient merely to snapshot a piece of news and associate it with a URL or DOI. We 
must understand the entire context of its technical acquisition and the temporal, geographic, 
social and personalized context in which it was seen and understood. Media scholarship 
requires different understandings of provenance and precision that differ from the 
opportunistic capture of traditional digital archival. This reflects the same challenges that will 
eventually confront the VR space. 


Interface 


More than a decade of collaborations with the Internet Archive’s Television News Archive 
has reinforced the centrality of interface to making archives useful. How something as simple 
as transforming linear video into a thumbnail grid can fundamentally change how we interact 
with it, opening the door to answering a wealth of previously impossible questions. Most 
importantly, however, the Visual Explorer suggests a coming confrontation between the 
richness of the unstructured VR space and the need to sample it into a discrete “skimmable” 
form. Video archives today are experienced as pointers into vast libraries: URLs that take 
visitors to individual videos or sections within them. There is no way to rapidly visually skim 
them to, for example, inventory particular metaphors or narratives. Prior to the Visual 
Explorer, there was no way to scan the linear form of Russian state television for depictions 
of nuclear blasts, appearances of maps, repurposing of Western media or the emotion of its 
framing. How might a rich and complex coming virtual world be similarly rendered 


“skimmable” and what does it even mean to “skim” an experiential reality? 
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Merging Human & Machine Intelligence 


The Visual Explorer reminds us that machines alone aren’t capable of the complex deep 
nuanced reasoning that lies at the heart of many of our most important journalistic and 
scholarly questions. Yet, no human could pour over a petascale video archive of millions of 
broadcasts. We need tools that merge our respective capabilities. What might that look like in 


the coming metaverse? 


Search 


What does it look like to “search” television news? What might it look like to “search” virtual 
reality? The reduction of its infinite richness to a common representational ontology in a 
metaverse must permit uniform search across virtual worlds with disjoint conceptions even of 
common objects, such as what “shoes” are, much as the globalized search of GDELT requires 
infinite interconnected contextualized knowledge graphs that represent the complete range of 
representations and experiences of a given concept. Even with the written word, how does 
one search for a concept that is precisely defined in the searcher’s language, but has no 


equivalent concept in other languages? 


Synthesis 


As large neural models produce increasingly humanlike text and imagery, our understanding 
of what it means to communicate and the linkage of text to image, the written and visual 
languages, is increasingly of focus. Today we can perform reverse image search to track all of 
these news images across the open web and cluster news to see how the same story is 
portrayed in different language and the captions of images. What will that look like in our 
virtual future, especially as machines increasingly can generate, rather than merely analyze 
information? As machines become increasingly adept at creating novel imagery, text, audio 
and video, what does machine “creation” look like in the virtual world? Merely the creation 
of code that defines a world, much as the creation of game assets and experiences? Or 


something entirely new that we cannot yet imagine? 


Dimensionality 
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News exists in spatial, temporal, cultural, cognitive and myriad other dimensions at once. It is 
created in a location, time and context, intended for other locations, times and contexts and 
consumed at yet other locations, times and contexts. An ordinary news article a year ago 
proclaiming the impossibility of war in Europe would have fallen into the obscurity of the 
ubiquitousness of its argument in the context of the world of that moment. This mundane 
article suddenly becomes extraordinary when rediscovered in the world of a year later. When 
we map the news in time, space, context and the myriad other lenses through which we 
understand information, we force that information to undergo countless transformative 
processes to project it from the moment of its creation to the moment in which we are 
attempting to understand it. This can be something as simple as representing a news event on 
a map in a geographic form with which it does not align, such using a city-centroid dot to 
represent a geocoded news article or a Place-coded tweet as a placeholder for a vastly finer, 
but inaccessible, geographic resolution. Or wondering aloud why an article was so widely 


ignored at the time when we understand its vast significance given hindsight. 


Interpretation & Emotion 


News is not just the conveyance of fact. It conveys emotion: of the author, of those being 
described and that which it creates within the reader. Yet, emotion is prefaced on context, 
community membership and lived experience, creating contradictions and complexities in the 
conveyance of emotion and how it should be understood. Such concepts are even more 
complex in time of conflict, such as wartime coverage lauding destruction and death as 
positive concepts due to their wartime utility. How is a sentiment analysis tool to adequately 
codify the emotion of a reader from a century ago when its underlying models were 
constructed based on readership and language use today? Few sentiment efforts focus on 
historical language use, given the field’s fixation on commercial deployments based upon 
modern language use. Moreover, sentiment models assume a single “truth” to the emotional 
conveyance of a given text, which belies the intensely personal and contextual notion of 


emotion. 


Transformation 


News is intended for human consumption through well-defined modalities of television, 
radio, earphones, screens, paper. How do we transform such material for machine 


consumption through the digital world? Those same complexities confront our coming digital 
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world creators in how to transform the world to VR into forms understandable by machines. 
Most importantly, in place of today’s codified lenses through which machines understand the 
world, what might a machine version of qualitative understanding look like? In place of 
today’s petabytes of JSON annotations, how might tomorrow’s AI models transform 


information into insights? 


Representation 


News is an imperfect and highly biased representation of the real world upon which entire 
fields of study are based to understand how those representative failures and biases influence 
both our understanding of the world and the functioning of society. How will tomorrow’s 


metaverse address such issues? 


In the end, the coming future brings with it not just a new world of human and machine 
interaction and new experiential mediums like VR, but reflecting back on the power of 
thinking differently about the information landscape, the future will usher in myriad new 
reimaginations of just what a “book” is and our ability to bring all of these coming 
innovations together to see the world through others’ eyes in entirely new ways never before 


imaginable. 


382 


Ken Perlin 


Closing Keynote: Experiential Computing and the Future of Text 


A decade from now, smart-glasses, and the networked infrastructure that will make them 
possible, will fundamentally alter everything from how children learn to how work is 
conducted to the meaning of shared public spaces. The reductive emojis of today's smart 
phones will give way to richer means of expression. It is hard to fully anticipate the impact of 
such a profound change on the nature of text, but we can make a few predictions. 

Perhaps the most fundamental long term change may be that text, and in fact language 
itself, will evolve to work together with physical gesture, because gestures will be able to 
make things happen in our shared computer-mediated physical space. The greatest agents of 
this change will be small children, because children seven years of age and younger are 
actually the creators of natural language. Once this technology is in their hands, they will 


evolve the uses and meaning of text in new and powerful ways that we can hardly imagine. 


Presentation 


2022-09-28 15:51:42 


Ken Perlin: So this is talk of experiential computing and the future of text. 
So my first experience is virtual reality. Probably. I've been thinking about this probably 
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happened when I was probably about six years old, and our neighbor who was a salesman, 
gave my brother and myself a set of. Plastic toy dinosaurs. And I would spend hours and 
hours and hours taking them on stories and adventures and making up all kinds of narratives 
with my dinosaurs. And then when I was probably about ten or 11, I discovered Harold and 
his purple crayon (Crockett, 2018). And that inspired me to think that, Oh, you can actually 


just create whatever worlds you want. 


And then when I was 16, I saw Fantasia, and then that completely expanded my 
consciousness and I said, “That's what I want to do when I grow up!’. And in fact their 


dinosaurs moved. 


And then about five years after that, I worked on the movie Tron, and I realized that working 
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with computer graphics in 1981 was not the same as what I had seen in nine made in 1940 


with Fantasia. 


So I started developing techniques to try to make computer graphics more interesting. I 
developed procedural techniques, what are now called compute shaders, the idea of running a 


complete program at every pixel, and that combined text and art in interesting new ways. 
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And then eventually those techniques made their way to making even better dinosaurs. This 


is a scene from Jurassic Park’. 


And so so then I joined NYU, where I worked on all sorts of things. I fled industry for the 
safety of academia. I developed the first Zoomable interfaces, which is apparently become a 


thing. 
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I did all sorts of crazy experiments, like I discovered playing with 3D printing. So I said, 
How could you print four dimensions? In fact, how could you print five dimensions? So this 
is a a tumbling hypercube. So I guess that's four dimensions plus one projected down to four, 


and it's a five dimensional object. 


387 


And then eventually, I founded a lab that was specifically about trying to look at what would 
be everyone's shared experience of the future (NYU’s Future Reality Lab). I was reacting to 
the fact that VR is a thing where you put on this headset and you go off into your own space 
and you're disconnected from the physical world around you and the people in the same 
room. 

This reiterates the historical experience of Edison's kinetic scope, which really didn't 
catch on, it wasn't until the Lumiere brothers put everybody in a big room with other human 
beings that movies became the dominant medium the early 20th century. It's because people 
really like to gather with each other. It's instinctive. It's part of our survival as a species. 

We did a whole bunch of experiments at NYU in which we put people in the same 
physical room in shared virtual worlds. Probably our biggest was in 2018, which we first 
showed at SIGGRAPH and then at the Tribeca Film Festival. We had 30 something people 
experience the same virtual theater piece together, but the people could all see each other and 
hear each other as avatars. And we showed that to several thousand people at SIGGRAPH 
and then 1000 more people in Vancouver. And the idea was to see what could you have as a 
VR experience that was socially shared. So it could be experienced eventually by hundreds of 
millions of people. 
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NOTABLE PAST PROJECTS 
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HOLOJAM IN FLOCK: A HOLOJAM DIA DE LOS HOLOS HOLOJAM DEMO 
WONDERLAND EXPERIENCE Don a virtual reality Participants appear as 
The first multi-audience, An interactive social headset and become a stick figures and see 
multi-performer virtual reality ghoul, ghost and each other as they 
live-action theatrical experience by David skeleton. Draw in air, create 3D paintings 
performance created Lobser, where play musical instruments together in virtual 
for virtual reality. audiences fly like birds. and dance with others! reality. 

Premiered: Premiered: Premiered: 

FoST Summit ‘17 FoST Summit ‘16 SIGGRAPH ‘15 


Well, meanwhile, in 2006, I had read Rainbows End (Vernor, 2011), a science fiction story 
that influenced me very deeply. The basic idea is it's 40 years in the future. Everyone wakes 
up in the morning, pops in their contact lenses, and they can see whatever they want. 

This might seem a little fanciful until you think about Gordon Moore's prediction, 
Moore's Law, made in 1963, which turned out to be quite prescient, which is that computer 


power doubles roughly every 18 months in one way or another. 


389 


° 
=\ongg 0 808s 
° 3° 
° 
2B Be 
Ca 
oo?? ° 
So ° 
. toe 
° 
° ¢ 
.e 2 
° ° 
° ° 
° 3 
z ° 
> @ 
° 
° 
°° o:4 
‘ ° 
° | aed 
o 8% 


And if you just start taking this and thinking about it, then this is what the future of VR is 
very likely to look like: 


And in fact, we can dive down into some details. Anything I can put on my head that's 
essentially like an Android phone is, which is what an Oculus Quest is, etc., etc. I can only 
get a few watts of power, but if I can plug something in the wall that's 300 watts of power, 


that's ten years in the future from whatever I can put on on my on my head. 
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Fortunately, Fast wireless is coming. We're only in the era of 5G now, but in another ten years 
we'll have 60. So we're going to go from from three gigahertz to 100 gigahertz. And when 
that happens, basically most of the computation is going to be happening not on your headset, 


but on something plugged into the wall. 


Internet 


Edge Graphics 
Computer coprocessor 


Edge Device 


So you'll be wearing some very, very lightweight thing that looks just like a pair of glasses. 


391 


micro-processor 


cameras 6G 
battery IMU transceiver 


2 \ 


6G base 
station 


speakers 


microphones 
displays 
and 
optics 


But not just the graphics, but the vision, the machine learning, the gesture recognition, the 
object recognition, all of the smart stuff is going to be happening basically in the cloud, and 
that's going to change the nature of reality. 
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We will have virtual objects that we just accept as part of the built world, and we won't even 
think about it just the way as creatures of text when we go to a restaurant now and open up a 
menu and we look at the text on the menu, we don't think that's amazing. We just think that's 
normal. Even though, of course, any non literate creature wouldn't understand why we're 
staring at cardboard to order food. Similarly, there will be creatures that will exist in the 
world and we'll just accept them and interact with them as though they're part of our build 


world because they will be. 


+ 


People will have face to face conversations in which whatever they want will be floating 
between them, and there'll be new kinds of interfaces that have very low cognitive load 
that will just be around us, instead of menus. 


There are dystopian scenarios. We don't want to recreate this for everybody. 
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We we want to have the ability to have calm interfaces as opposed to just say when I have to 
put on my glasses, just as today people have to have their smartphones and yet everybody is 
advertising at me. So ideally (by the way, everything I'm showing you is a live demo), I want 
to be able to just sketch out a creature. 


And the fact that I drew that creature means the creature is in my world, maybe wants to eat 
my plants, interacts with things, etc. and we want to have that kind (Ed: of interactions)... 

I was heavily influenced by reading in 1990, 93, Steven Pinker's book, Language 
Instinct (Pinker, 2003), where he pulls together work by many, many people on 
computational evolutionary linguistics. So, for example, we learn that children up to the age 
of seven invent language ,languages evolved by children. And when you think about this, it 


makes sense because if anything, that's not learnable by children, up to seven cannot be 
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passed on, so children actually evolved language, not grown ups. 


And then I learned about things like Nicaraguan sign language studied by (??) Cengiz 
and others, where you see a generation of children, deaf children evolve grammar before 
everyone's eyes, learning how to create more interesting visual constructs to create more and 
more complex re-combinatorial syntax, which is only found in natural nature and natural 
languages and DNA, as one of the earlier speakers pointed out. I started playing around with 
these ideas. What would a future visual language look like? 


And this is this thing called ChalkTalk, where I basically say, okay, so you have this idea of 
nouns, and because the nouns have a certain quality, they move. But, but maybe the way you 
draw this thing, so I'm drawing this live now changes the way it moves and. You can tie 
things together and ask questions. I use it to teach science and computer graphics. So, for 
example, here, this is a pendulum. And I can also find out like, Oh, what is it about this 


pendulum that is interesting? And it's that it has actually a sort of decayed sine function. 


But notice that again, in terms of adjectives, if our adverbs, if I draw this thing differently, it's 
the same object, but it now has different physics and I can tell entire stories with this. So, for 
example, let's say I wanted to talk about energy conservation, so we have an idea of light. We 
have an idea of of a motion sensor, and I can tie the motion sensor to the light. And this is the 
part of the talk where I do a little bit of hand-waving so I can have a hand. And if the hand 
moves in front of this and all that's going on here is code that anybody can edit. And as you 


edit this code, different things happen and you can create different sorts of objects. 


And just to sort of sum up, by the way, one nice thing is that computers are now 
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millions of times faster than when I first developed procedural textures. So now these 
procedural textures can happen in real time. This is the same sort of texture I made to make 
that marble vase, but now it's running many, many times a second and people can do real time 


design with this. 


The software hasn't changed. It's just that the computers have gotten faster and faster 
and faster. So to sum up, I feel as though language is going to evolve. And I think about little 
six year old me. And in the future, when kids are able to create and evolve language using a 
visual component, what's going to be normal, everyday reality is going to basically be like 
what we today might think of as Harry Potter meets Harold and the Purple Crayon. And I'm 
hoping that we can all help to make that future of text happen. 


O&A 


Alan Laidlaw: I've seen you demo ChalkTalk Ken, many years ago and I love it. And it's 
been a probably a cornerstone of what I try to build towards and think about. It's interesting 
in the context of GPT three, seeing the demo again and realizing, Oh, this is like prompt 
engineering before prompt engineering, which kind of got me thinking around. The. Do you 
imagine a version of to a general audience of drawing with Chalk Talk? But the response, 
the translation is wrong. You know, the ball, the pendulum is not what you had in mind, 
right? Would would the. Would there be a way to I guess in the daily world, you could say 


like regenerate the image, try again. What are the other possible near matches? 


Ken Perlin: Well, I think language is a funny thing because and this is hard for me to wrap 
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my head around as I started thinking about it. We are all experts. Children are learning 
machines that are specifically good at learning natural language like language. Natural 
language is by definition, the thing that children learn really well. You try to teach kids 
Esperanto and they will spontaneously fix it because it's not a natural language. It doesn't 
match the way their brains want to learn. And so what Chalk Talk is trying to do in a way, is 
suggest when you have whatever is the future language, that there then what will be the 
feeling of that kind of conversation? So clearly, I'm I'm an expert at this. You know, I know 
what the vocabulary is, but I'm trying to imagine a world where everybody is in on the 
conversation, which is astonishing, is that every single person on this conversation can 
spontaneously, with no cognitive load, put together a grammatically correct sentence that's 
never been uttered before. And we just take that as the the base of human experience. So I'm 
not too worried about people making mistakes and having errors because that's part of how 
people talk. You know, somebody said somebody forms a sentence and creates an accidental 


one. That's just part of the common. 


Alan Laidlaw: So just a quick follow on on that then, because that's interesting that using the 
metaphor of speaking and creating a language and language evolving 100% on board with. 
When it comes to drawing the language, writing it out, the just, I guess how much is a. How 
much there is our hand, a kind of intonation versus a kind of language? Like when we 
sketch something, we we all sort of sketch arrows differently, right? Is that like a figure of a 
way of speaking that differentiates us? Is it kind of like a voice, or do you see that the hand 
being an essential part of. This new evolved language. Does that make sense? 

Ken Perlin: One of the things that I do know a number of people have studied this is that the 
centers of our brain that control hand manipulation are very strongly tied to the parts of our 
brain where we use the word articulation for both verbal language and our hands. And in fact, 
there are some there are theories that the parts that are the language parts evolved out of the 
parts that are used for prehensile manipulation. So and in fact, everybody that you watch that 
speaks, they just automatically start moving their hands. So there already is a very strong 
connection. And I think if you look at the the beauty of what's been created and not here in 
communities with sign languages, which are incredible, I mean, they're just there's this there's 
this wonderful power of simultaneity that we don't have as serial speakers that I think we can 


all move toward that. 


Fabien Benetou: I remember I tried ChalkTalk , I think, in 2019 when you presented it at 
NYU and released it on GitHub. And I remember cardboard mode, but I don't remember 
trying it in VR. Namely that you would, let's say, pinch in the air to start to do the same 


shapes and then be in immersive mode, even though it is web based. I'm wondering, has it 
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have you tried that before? If not, it not. Wasn't it interesting? And if you haven't tried it, why 
not? 

Ken Perlin: Well, we've ported in our lab at NYU. We've ported chalk talk to shared socially 
shared VR, and we just look at it as one of the aspects of the many, many research questions 
about sharing virtual and extended worlds together. We don't focus entirely on the language 
question, but it is one of the things that we look at. So it's in there, but we also look at, for 
example, asymmetries and scale. How do you how do you use virtual characters as agents? 
And they're just lots and lots and lots of the relationship between tangible objects and their 


virtual proxies. But one of the things we do look at is gestures as semantic creation. 


Brandel Zachernuk: Thank you for the talk and for the presentation on ChalkTalk. I'm 
curious as to the sort of the extent of the utility that you found in real experiments with 
Chopped talk. I recall Ivan Sutherland talking in the nineties about sketchpad in the in the 
sixties, saying that ultimately it only really had two uses. One was to present the graphics for 
his thesis and the other was to draw some hexagons for his mother. And he never asked why 
she wanted them and the limitations being that a lot of things need representations or metal 
representations that that he wasn't able to come up with a graceful representation for. How do 
you find how have you tried and how do you find scaling chalk talk and what ends up being 
representational and where it ends up software? 

Ken Perlin: Well, the big secret to ChalkTalk is that and this I got this insight from from I 
got to meet my my one and only hero who just like Ozma which is Randall Munroe who does 
xked and he I asked him what he thought about and he suggested a chalk talk and he said, 
draw the simplest possible thing. So what I found was the big insight was I go up to I want to 
say, Oh, I want to draw a planet or I draw an A person or I want to draw a duck. I go up to a 
whiteboard and make the simplest possible drawing, and I find that that's the right visual 
representation to start. And behind that I can put whatever code I want. I mean, I believe 
sketchpad was very pure. It was doing everything. I mean, I've been doing everything 
through this very pure system of constraints, and Chalk Talk is really a hodgepodge of 
techniques. It's it's really a way of, of, of sort of having an interface without apps so that 
instead of here's an app and here's an app, everything can talk to everything else. So I would 
I'd say I'm not running into the same limitations he was running into because I'm not trying to 
make something pure. I'm just basically trying to make something to present ideas for people. 
And if I have a new idea, I program it. I come up with some simple representation. I have 
ways of morphing things, and I think he was going for something much more idealistic in 


1968 than what I'm going for. So it may be a little apples and oranges. 
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Andreea Ion Cojocaru: Hi. Thank you so much for this. This is wonderful. Are you thinking 
of this tool more in the sense of a cognitive tool or a communication tool? And I'll try to 
qualify that. I've been reading a bunch of Barbara Tversky studies and two of them come to 
mind that might be relevant in this context. One, she, she, she, she found out that when 
people are presented with a process or a mechanism that they need to understand, when she 
asked them to use gestures to mimic the movements of the process of the mechanism, they 
learn better. And also, when she asked them to sketch out the steps involved, they also learn 
better. But she also has a study on animations, and she actually found that where people were 
shown an animation when all the parts are moving, they it ended up not helping as much as 
they thought they would. So there seems to be something about either the internal movement 
of the mind trying to understand or a movement of the body trying to help the mind move and 
understand versus the low hanging fruit of just being given the movement that seem to be 
quite advantageous in terms of a cognitive approach, a cognitive perspective. And I believe 
Barbara is in the audience. So I this is more also a prompt for her to to step forward and 


correct my interpretation of her work. Thank you. 


Ken Perlin: Yeah, I'm a big fan, of course, of Barbara Tversky's work, and I've learned a lot 
from reading her papers and having conversations with her. And I think it strongly informs 
what we do. As I mentioned to the previous answer, Chalk Talk is one of the early artists. 
Chalk Talk is just a component of the kinds of questions we like to ask in our lab and the 
question of embodiment and in fact, shared embodiment between multiple people is really 
important. And all of her work, not just on that you cited, but also the use of dictates. Like 
when I'm in the same room with somebody as she and her students showed, you can say this 
or that or then or refer to things implicitly. And because things are embodied and you have 
gesture and eye gaze, etc., you're able to use language in this more implicit way, which is 
very, very powerful. So we're hoping that everything that we do will lead to a trajectory 
where people are in their bodies and using the full power of language which is meant to be 
used together with evolved to be used together with the physical embodied human in the 
same room. And I. Temporarily being stuck on Zoom is doing a terrible disservice to our 
power as humans, because who am I looking at right now? We are really, really good at 
integrating language with attention, direction, and I think once the technology catches up to 
the studies that she's doing, we're going to get a lot more out of these communicative 


technologies. 
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Livia Polanyi 


Virtual Vision 


I have bad eyes. They don't focus together and reading is difficult. I read almost exclusively 
on my tablet now which limits my choice of material largely to e- books. Very few scholarly 
or academic books are available for the screen and they are expensive especially for people 
without access to a university library system. 

At the moment, propelled by a nagging curiosity to read about the work of the 
psychoanalyst, D. W. Winnicott, I have broken out of my digital prison and am currently 
deeply immersed in the messy theoretical and personal wars that roiled British 
psychoanalysis for the first half of the last century. Currently, I have ordered a “marks of 
paper” volume about the life and work of Melanie Klein, a formidable force who led one of 
the main combatant forces in the conflict to increase my understanding of what went on and, 
while am awaiting delivery on that book, I am breaking up my reading about Winnicott with 
quick dives into a volume of short biographies of Freud's patients— whom he seemed to have 
been of no use to— and a hard cover dealing with obsessives who collect 78 rpm records. 

So here I sit, in my oversized comfy brown leather chair, with a small pile of books that 
is about to grow larger strewn about within reach of my arms. One volume on my lap, 
perhaps, another on a table to my left, the third wedged in between me and the side of the 
chair. Finding where I broke off reading one when switching to another is always a bit of a 
hassle and locating a delicious quote to share with someone else requires fumbling around 
and frequent giving up before I locate the titbit. Of course, my tablet and phone are close at 
hand, too, since I need to chase down references, read up on articles I can get access to in the 
web, consider buying another book, get lost in some side path triggered by something or 
other and, of course, capture images of particularly interesting, enraging or downright silly 
passages I come across in my sedentary voyages across various landscapes. 

So, what does this all have to do with text, knowledge, XR? Well, quite a lot actually. 
Allow me for a moment to propel myself into an XR future scenario. I am once more at work 
pursuing knowledge, the books I am reading strewn about my digital chair. I move from one 
to the other and from the books to other digital resources effortlessly, simply asking for which 
one I need now or maybe merely searching out the volume I want with my eyes. Following 
up on a reference, a question, an intuition is a snap— I merely request more information and it 
appears— similarly notes to myself I might want to make or messages I might want to send to 


others can be composed merely by asking they be created. Those memoranda can easily 
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include all the information I or my correspondent might need to access the sources 
mentioned. Capturing texts and creating linked files or other representation of exact 
quotations, relevant images, sources consulted or to be consulted and even snarky comments 
appear almost with the speed of thought. Editing, changing, deleting— are effortless. 
Physically still seated in my comfy leather chair with a cup of non-digital tea nearby, my 
mind roams freely through an imaginary library, filled with digital tools and resources, 
trusted amanuenses and tireless creators of indices and notations that allow me to wander 
through the fields of knowledge whether from psychoanalysis to vintage recording collection 
or from any topic to any other topic where my mind and poor sight want to go. 

While physical books are comforting “transitional objects”, I look forward to roaming around 
a virtual library, my personal reality augmented by emerging technologies. While seated in 


my chair, drinking my entirely real world cup of tea. Unlike Captain Picard, however, I will 


prepare my tea ina real kitchen. It will not be Earl Grey. 
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Lorenzo Bernaschina 


Gems 


Gems is a personal knowledge management tool to explore and connect ideas visually with 
the help of AI 


The total amount of information in the world is growing exponentially. Information overload 
is everywhere: on media, in companies, at school, on both our physical desks and digital 
desktops. There is a hyper-production of content and many contradictory sources available. 
Finding signals in the noise is becoming increasingly challenging and expensive. In 1982, 
Richard Buckminster Fuller estimated the knowledge production rate of humankind. In his 
book "Critical Path" he described the "knowledge doubling curve" by explaining that the rate 


at which information doubled was getting faster and faster. 


Today knowledge workers are drowning in information they don’t have time to process. 
We save interesting web articles and social posts we rarely revisit. We have messy desktops 
and folders. We have many books we barely have time to read, let alone interpret and digest. 
Same for newsletters, videos, podcasts, PDFs etc. 

To make sense of this flood of information and make use of it, we have note-taking 


tools and cloud storage services that share a common design pattern: 


¢ They are built around hierarchical file directories (folders) which are the digital equivalent 
of the filing cabinets or shelves we had before the existence of computers and software. 
Back in the early days of personal computers and GUIs, designers needed to make them as 
familiar as possible to users. So they made skeuomorphic adaptations of the Desktop, the 
Trash, etc. Similarly, it felt natural to keep digital information organized as the physical 
one. It was easier to be adopted by users but the drawback was it didn’t leverage the full 
potential of the new tool. It’s hard to keep information updated, create cross-references, 
find patterns, surface ideas, or even just find something in filing cabinets and shelves 
(the Zettelkasten of Niklas Luhmann is probably the most audacious endeavor in this 
regard). These limits are reflected in file directories. 

* To overcome them, they introduced tags and metadata, so we can attach semantic meaning 
to each note or file. However, it is manual work that requires a lot of cognitive effort. It’s 


time-consuming and the larger the knowledge base, the harder it is to keep it consistent 
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over time. We have to set conventions that are difficult to follow because sometimes we 
forget to add a label, sometimes we say “I'll do it later” (which is never), sometimes we 
use synonyms or slightly different spelling, etc. 

¢ For many of them, the primary interface is a text editor. This forces us to think linearly 
within the boundaries of the page. It’s hard to see how the moving parts of a learning topic 


or project come together from there. 


Because of these design choices, existing note-taking tools and cloud storage services act as 
passive storage containers. We create a note or save a file, add some tags, and leave them in a 
folder that we rarely revisit. We have to remember what kind of information we saved in the 
first place. We have to know what keywords or search methods will allow us to find it again. 
We have to develop ways to integrate old ideas with new ones. We have to prompt ourselves 
to expand on our notes, combine them, synthesise them into new realisations, and critique our 
own conclusions. None of these tools is designed to help us draw associations between 


concepts, which is how we really make sense of the world and generate new ideas. 


Gems is designed to encourage associative and non-linear thinking through a combination of 


graph visualization and artificial intelligence: 


¢ The main interface is an infinite whiteboard. You can import, create and visually arrange 
notes on it. You can group notes into nested layers as in folders, but keep visual references 
between them with connections. So you have a flexible structure that matches your non- 
linear way of thinking, to build networks of thoughts instead of siloed containers that 
don’t talk to each other. 


* On top of it, AI constantly indexes your knowledge base. If you need to observe notes 
from non-trivial angles, such as semantic similarity, you can ask the AI to do it for you. 
You can make sense of retrieved results in dedicated views and make updates from there 
(e.g. connecting two similar notes) which will automatically reflect in the knowledge base. 
Once you return to the main whiteboard, you will see how the updates fit into the rest of 
your knowledge structure and you can rearrange some of it accordingly. This helps you 
review your overall understanding of a topic in light of the new insights suggested by the 
Al. 


If you are a lifelong learner, for example, you can visually build a map of concepts from your 


readings and ask the AI to suggest connections between them. If you are a non-fiction writer, 
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you can keep track of all the sources and use AI to get a list of the most relevant to write an 
outline in a fraction of the time. If you are a fiction writer, the whiteboard helps you build the 
narrative world of your story visually, define characters, places, events, and see how they 


come together in your plot. 


Artificial intelligence and human intelligence ultimately solve very different classes of 
problems. Machines are very good at processing a huge amount of information fast. We are 
very good at finding meanings, generating new original ideas and making connections 
between them. The magic happens when we combine the two. Gems captures the semantics 
of both your brain and AI, the former through the digital whiteboard, the latter through 
sophisticated large language models, and makes them communicate together harmoniously. 

Computer science was born with the promise of extending the human mind with 
technology. Personal computers have kept the promise and now the technology is ready to 
take that bold vision forward with AI. That’s why I focused my studies on it after graduating 
in software engineering. Gems brings this power to creators, educators, researchers, 


journalists, and any other knowledge worker. 


If you want to be part of this journey, please visit https://gemsnotes.app/ 
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Mark Anderson 


Image Maps and VR: not as simple as supposed 
Abstract 


Although it might be supposed that interacting with infographics in VR is ‘just’ a matter of 
using existing 2D image maps, it turns out to be less straight forward. Here, a few of the 
unexpected issues are explored with implications both for human users and the tools they 
employ in this context. The user’s methods and their tools both need some improvements to 
make the most of these new opportunities. Bolded text indicates sections of note for the skim- 
reader. Re-using infographics in VR need more effort than merely adding a simple image 


map. 
Background 


The Future of Text (FoT) weekly discussions have included exploration of interacting with 


infographics in VR, using Bob Horn’s various murals7® as the initial subject matter. Doing so, 
it became clear that a number of issues associated with that process are not well integrated, 


tool support is poor and that more explanatory documentation would be helpful. 
The Problem Space 


Whilst this exploration started with the above murals, the process is actually generic to 
moving any infographic from 2D to VR/AR use. Within that, there are two types of 
presentation to consider: bitmap/raster vs. vector graphics. Not considered here is the further 
complication of static displays based on dynamic data (static render of a dynamic source). A 
further issue is a degree of mismatch of the 2D pixel concept and 3D modelling methods. 

It is reasonable for the casual reader to not care, personally, about the differences of the 
2D vs. 3D/VR medium. But for those intending to move artefacts from one to the other—or 
create artefacts for such re-mediation, the differences of the two media’s design methods and 


formats do affect re-mediation in VR. 
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Display in 2D and bitmap (raster) vs. vector formats 


On a 2D screen a pixel originally described the smallest discretely addressable part of the 


screen, an area that can hold a discrete colour value or pixel (explained?’). In a bitmapt 
image each the image grid maps 1-to-1 with a pixel. In reality, it is far more complex than the 
simple physical grid we imagine partly due to constant improvements in displays, leading to 
notions like the ‘CSS pixel’. 

Vector artwork defines an image in terms of a series shapes that can have strokes 
(borders) and/or fills. The vector approach makes the image independent of any particular 
resolutions (i.e. pixel size). However, such artwork is almost always rasterised on-the-fly to 
display it on a normal 2D display, though it allows scaling without loss of clarity (if scaling is 


applied before the image is rasterised). 
The (HTML) Image Map 


It is useful to describe the image map in the context of the Web for two reasons. Firstly it is 
the context in which the reader is most likely to have met the concept. Secondly, discussions 
in the FoT group have suggested that Web, or Web-compatible, standards will be important in 
how our work may move to/from the VR environment. 

An image map defines (non-overlapping) shapes within the area of a webpage 
apportioned for a given image. The aim is interactivity: clicking on map area A opens link X, 
whilst clicking on area B opens link Y, etc. Thus one map may contain links to many different 
resources. 


Image maps have been with us since the very early days of the Web. Insertion of 


images in web pages were first proposed*? by Marc Andreessen in early 19937! and shortly 
after Mosaic added an ‘<ismap>’ elementt which was essentially the first image map, 
implemented server-side. Even then, Tim Berners-Lee had noted that whilst fine for 
k>?—though the latter 


was not used natively in Web pages at the time. Though current web browsers can now 


bitmaps, this mapping method was less useful for vector artwor 


support the vector SVG format, crucially, the image map areas (shapes) are defined in pixels 


as offsets within the host images declared display size. 
In 1997, the server-side ‘ISMAP’ concept was adopted as the W3C HTML v3.2’s 


client-side ‘map’ element>? and it lives on into the current W3C HTMLS5 specification”’. 


Initially popular, especially for page navigation sections, push-back against use of text 
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embedded in images saw interest move on. As a result image maps, implemented in HTML, 


are encountered far less often. 
Pertinent too, is the fact that software tools had to adapt fast to add features to make 


map mark-up easy; Adobe had to quickly develop a whole new tool ImageReady*> (later 
subsumed into Photoshop) and similar happened for other vendors. This problem of a lag of 


affordances for new uses in our creative tools continues, as elaborated below. 


Raster vs. Vector Data 


In the early Web images were all raster graphics>®, those using the 2D ‘pixel’ grid, were the 
only graphics supported. This is the type of data for which the above image map was 


envisaged. 
Much more recently vector graphics>’ arrived in the form of SVG*®. Another visualisation 


form is the HTML <canvas> element?” that uses JavaScript to draw sha es, in a vector-like 
p p 


manner but essentially results in a rasterised display. Most recently we have the likes of 


WebGL that can draw shapes in 2D or 3D*?. Whilst these methods support embedded 
interactions there appears to be no consistent notion of an image map. Unsurprisingly this 
means that relevant image creation tools lack affordances for making ‘image’ maps. 

Whilst static infographics (i.e. with no dynamic elements), like the murals above) can 
be brought into a VR space and displayed, they offer little further affordance unless within 
the context of a web browser object. There is no simple and consistent way to interact with 
the data (of which more below). These ‘dumb’ documents were designed to be displayed and 
read but not for digital interaction, especially in a VR environment; this poses a challenge for 
digital enrichment and re-mediation. Static images/charts have limitations for easy VR 
enrichment & remediation. 

Simply displaying the graphic in VR, as if a painting on a wall, is comparatively 
simple. The harder part is being able to interact with a particular element—or a set of 
elements—within the image. This might be to explore the sources of an annotation or the 
issue it addresses. Or, it might be to re-present content in a different type of view, such as ina 
timeline. Consider too, that the source image—or its VR frame—will potentially be 
folded, zoomed or skewed in a manner that displays the graphic differently from its 
normal 2D display; this may occur either to display it on the surface of a 3D object or to re- 


mediate the content into other visualisations. 
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Issues for Presentation of Infographics in VR 


Given the newness of VR, unsurprisingly the larger amount of infographics we might wish 
to bring into VR today predate notions of such use, so it is useful to consider legacy files 


from those we may create today or in the future. 


Displaying image data in VR 


Here, methods are less well defined than for 2D, reflecting the newness of the medium. An 
important point is to understand that 3D/VR is not created using cubic pixels, i.e. a direct 
extension of the 2D pixel. Image data, such as infographics, will normally end up as a 
rasterised fill laid onto the surface of a 3D object; this potentially removes some existing 


advantages of vector formats (in 2D). 


Even if displayed in VR like a picture hung on a wall, the ‘picture’ is still part of a 3D 
object—albeit of tiny depth—so skeuomorphic 2D descriptions can be unintentionally 
unhelpful: skeuomorphic descriptions help describe the visual experience but can 
obfuscate how it is constructed. 


All surfaces are not web displays 


One way to display existing images is in an object that holds a web browser object, but that 
then interposes another whole layer of structure (the web browser) to ‘just’ display a picture. 
If the display has a dynamic element it may be useful—indeed necessary—in the short term 
to use an embedded browser object. Yet, if we wish to interact with an infographic, must the 
targets of the interactions be an endless growing collection of browser objects? If so, we 
should give attention to lighter ‘weight’ web objects so multiple use doesn’t generate unseen 
and unwanted overhead. Do we always need a ‘full’ browser object to display a usable 
HTML image map? 


What is to be linked and where will the linked resource be found? 


The HTML image map assumes a (click) interaction loads a URL. Originally, those URLs 
would have been web pages but now might be any valid resource such as a query-driven 


dataset relating to the clicked source. A question to ask, given the FoT group’s focus on 
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Visual-Meta and local resolution, is that—for new documents—what of the linked data 
travels in(side) the mapped artwork, or as a local but discrete (‘sidecar’) resource, or simply 
uses a URL and trusts to the current environment to de-reference it? How much re- 


mediating data should travel in/with the main image? 


Considering this has implications for how data (transfer) format should evolve to 
support movement into and out of VR, as for a while work will likely involved mixed 
environment use or AR. Additionally, the type and range of environments my vary by 


participant in a shared workspace, so a ‘one size fits all’ is over-optimistic. 


Hitherto stable text formats like RTF?! (or text+image RTFD*) served the paper age 
well but their utility is lessened when re-mediation benefits from access to the document 
structure and style information is stored separately from the text, making it easier for 
different media to style optimally for that medium whilst reflecting the spirit of the author’s 
intent. RTF hides that relationship. Although RTFD stores image data outside the RTF 


stream, the intermixing of style and content in the text still remains.. 
Whether data always need be strictly Web-compatible is not yet clear. But the Web’s 


notion of a ‘DOM’ (Document Object Model**) is useful—the exposure of a text’s 
semantic structure. Whether only for anchoring visual styling or for allowing more complex 
interactions, a DOM—or similar structural description—clearly offers more in the VR 
environment than in 2D. In 3D, we are less strongly bound to manifesting essentially 
Print-era presentation and may more readily move to more complex interactions and 


reconstructions. 
Legacy Files—re-mediating pre-existing resources 


For raster images, using a Web (HTML) ‘frame’ to hold an infographic image is a tractable 
approach for creating discrete interactions but the frame object requirement may limit the 
ability to do much more than display/scale the image. Plus there are the scale issues of using 
multiple such object, as already discussed. Unless the source is high resolution, a paucity of 
pixels may also limit effective transformations in VR. 

For vector files the image will, at present, likely be rendered in raster form even if from 
a vector source. So the click event anchors directly to the HTML defined area. Less clear is 
whether direct interaction with SVG embedded in a web object offers an advantage. For 
instance, it is also unclear how the SVG click event responds if part of the parent image is 
folded (i.e. hidden). Most likely, in a web frame context, this will be down to the browser 


object rather than the VR object on which it is displayed. Thus for some transforms and extra 
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level of complexity is added: to transform the 3D object, the limitations of the embedded 
browser must be acknowledged. 


Current files—content designed for combined 2D/3D use 


For pre-existing images little changes for raster files as the HTML image map remains in the 
HTML specification. By comparison, vector artwork mapping could improve considerably if 
creative tools were to add clearer tools for marking/mapping images and making it easy to 
connect the right data but that may be over-optimistic. 

As vector artwork offers greater scope for transformation, also open is the nature of the 
the likes of the SVG click event (as discrete from an HTML image map click). Rather than 
simply point to a URL as in the past, the event might actually trigger a visual 
transformation, reveal extra information, etc. 

Indeed, being able to ‘paint’ the SVG data more directly onto a VR object without a 
‘browser’ layer would offer a less complex interaction moving in and out of VR, even if only 


in the volume of cross-environment traffic. 
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The nature of VR interaction 


Even having imported an infographic into VR, drawn it onto a suitable object, we now need 
to consider interaction. Due to the way an infographic may be rendered onto a VR object, we 
may know what we wish to ‘click’, but doing so may not be so easy. A busy graphic may 
require fine-grained interaction to reach a specific point of interest but that must nonetheless 
match the fine-positioning ability of the interacting agent (human or otherwise), or else the 
degree of interaction is impoverished. 

A useful affordance of VR is to give a limitless screen allowing for large changes of 
scale. Therefore, if interactive elements are tightly positioned within a 2D design, there needs 
to be some sort of metadata to signal the creator’s intent as to what—if anything—becomes 
an interaction target when the granularity of discrete targets is finer than that of the 
interacting agent. To do this requires a means to the creator of the infographic to define and 
store that information; such features do not yet existing meaningfully in mainstream creative 


tools. 


Tool support for linking and re-mediation 


Creative tools do seem to be a current constraining factor. These are not the tools used by the 
prototypers at the leading edge, where the tools themselves are evolving. Rather, the tools for 
the ordinary creators who represent the larger volume of such creative work. For instance, 
when a new infographic is planned and which will have a lot of mapped (linked) resources, it 
would be useful to be able add a pre-structured grid of links or per-VR-addressable-item 
layers. In parallel, it may help to have methods where more complex data is simply bound via 
a GUID, and the GUIDS mapped with to either a grid- or layer-based document structure. 
Such VR-mapping-inclusive thinking does not yet seem present in the design of large 
scale creative tools. 

However, the example of early Web graphics offer a clear example of how change is 
problematic for established genres of creative tools. New features can be added but this is not 
necessarily optimal for the user. When adapting for new methods it is not always optimal 
to simply try to force new methods into old processes. This is a challenge, because for new 
environments like VR, the necessary feature set for tools is not yet defined. The experimental 
nature of prototyping means that it may only hint at such features—aunless the prototyping is 


intended to codify new processes and the tools/features to service it. 


Lest some of these new tasks seem trivial, take as an example Bob Horn’ s mural? on 
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the UK’s nuclear waste program. Whether in vector or raster form, the discrete textual 
elements alone number over 450 (i.e possible discrete click targets), without even addressing 
some of the purely pictorial with which the VR user might wish to interact. Consider the task, 
today, of defining each target, manually, and its associated target data. This shows we need 
tools to allow intake, into an otherwise purely creative space, of structured data that can ether 
be used to scaffold the infographic creation (i.e. one object per layer). Alternatively, we need 


a means to rapidly attach data to each large numbers of objects in the source file. 


Even with such new tools, user education matters—we can’t just assume everything 
is intuitive or made usable by ‘someone else’. The gap between imagined exploratory re- 
use of existing infographic data and what any but the expert may achieve remains large. For 
the person trying to use such material in VR, there is a human issue of education: both 
understanding the implications of thinking beyond legacy static publishing notions and an 


interest in and learning of the tools that can deliver a richer VR experience. 


Conclusion 


If our temptation is to think “we will just use an image map” as the process for infographic 
display in VR, it suggests that in our rush to imagine VR working we aren’t also taking time 
to consider the emerging process to render our imaginings. It may be the case that image 
maps are a part of the solution but aren’t the complete answer. Why so? An image map with 
appropriate data, is only a help if the rendering tool can understand it, whilst even 
appropriate data is insufficient if the human user doesn’t fully understand creation of 
that data or know which tools if any they can use to make or manipulate the data. 

Even if all or most VR objects are essentially browser-type displays, an image map 
with appropriate enrichment data is only a help if the VR environment can render it, whilst 
retaining the ability for appropriate interaction. Alongside this metadata is insufficient if the 
human user doesn’t fully understand the how to structure the data for interaction nor has the 
tools at their disposal to do this other than manually. 

Thus it is that (re-)using infographics in VR go beyond the current notion of the thing 


we call an image map. 
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Matthias MUuller-Prove 


On Real and Virtual Text 


Naomi silently moves her tongue with out opening her mouth. The MSR sensor — the 
mumbled speech recogniser — on her neck detects her intent and opens a matrix of chapter 
previews. She points with her finger in the air. Then she opens her hand. The room dims 
down while transforming into her preferred reading environment. Naomi has smart-designed 
this room according to some old photographs she got from her grand-grand-grandmother Isa 
Bowman. She begins to read: } 


From Language to Text 


Language is a well-formed sequence of words to express thoughts and ideas. Spoken 
language is linear. Spoken language can be turned into text by writing it down. Text is linear 
to the extent that it consists of rows of words, separated by automatic line feeds at the margin, 
or by hard carriage return control characters to give way to a new thought in the following 
paragraph. Once a sheet of paper is filled up, the words continue their journey on the next 
page... until this is full... and so on... The sheets of paper pile up to form a book. A book is a 
physical object in real life. 

Books are a natural habitat for text — same as magazines, newspapers, reports, hand- 
written letters... basically all paper-based media. Before capturing the messages on paper or 
papyrus our ancestors used to impress clay or carve in stone. A few thousand years later we 
use invisible magnetic or electronic charges as computer storage and memory. Each charged 
physical spot represents a bit, a binary digit 1 or 0. Eight bits to the byte and a decoding 
convention like ASCII or Unicode — these are the basic principles to interpret the bits as 
characters and to display them with glowing pixels on screen. All three modalities of text — 
pre-paper, paper, digital — are still in use today; for instance (i) on gravestones, (ii) for the 
classical publishing industry, and (iii) for all kinds of computer media from personal word 
processing to social media. 

Text is linear — thinking is not. Language has the expressive power to put complex 
ideas into words by utilising its meta-referential properties. This enables an author to directly 
approach a reader and point to certain sections of the text. Complex causalities or abstract 


ideas can be described and discussed with words. New concepts or things can be handled by 
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assigning new names, and by putting them in context with familiar terms. In fact it is quite 
difficult to find words that are not metaphorically derived from prior words. Quotes are often 
used to indicate that a word is not meant as such but shall be understood in a metaphorical 
sense. ‘Virtual’ is another attribute to inform readers that the following word should not be 
taken literally. We will discuss “virtual reality” further below. 


A discussion among several people can be captured with linear text — as long as they do 
not speak at the same time. If they do anyway, we would either need a multi-track score like 
music notation for the instruments of an orchestra, or the text itself explains that the 
following sentences are meant to be spoken simultaneously. That would be an example of 
written language’s meta-referentiality. Footnotes are like a second track as well. They are 
anchored to the main body, i.e. a little spatial hint which indicates when reading the side track 


might be intended and appropriate. 


From Text to Online 


Hypertext — a term coined by Ted Nelson in the early 1960s — is non-sequential writing. Text 
passages are individual units. They can be connected by hyperlinks to provide related content 
to each other. Each link bears the invitation to follow a different, but somehow connected 
thought though a rabbit hole. {Naomi smiles.} Quotes and references are a primary citizen in 
hypertext because the origin can alway be accessed in its original context. Link and reference 
structures are visible on screen, e.g. as lines or coloured shapes between related text sections. 
[Ted Nelson, 1972: Parallel TextfaceTM in XanadouTM in Matthias Miiller-Prove, 2002: 
Vision 

and Reality of Hypertext and Graphical User Interfaces, section 2.1.2ttt] 60 years after the 
idea of hypertext, the online environment is not a dream come true. 

On the pro side, the Internet is a common communications infrastructure connecting all 
continents. It delivers all kinds of data and services to each point on the planet. A tremendous 
success and innovation which shall be used in a beneficial way for all of us. However... 

The Web as we know it today has almost nothing to do with the original vision of an 
interconnected dynamic global library. The only link between hypertext of the 1960s and the 
Web of the 1990s are hyperlinks between Web pages. Even Web 2.0 is history already. Web 
2.0 was a term made popular by O’Reilly’s Web 2.0 conference series in the 2000s. It is the 
shift from tech-savvy or professional website creators to average people who want to upload 
“user generated content” and edit their personal pages. Web 2.0 is the beginning of a 


democratic medium where everybody can participate and easily edit wikis and write blog 
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articles for the interconnected blogosphere. Since the 2010s big tech and media corporations 
rule the market, for instance Meta (facebook, instagram, WhatsApp, Meta Quest 2 and 
presumably the Metaverse), Amazon (Kindle, Echo, Prime, AWS), Apple (Mac, iPhone, 
iPads, Watch, podcasting, TV), google (search, YouTube, Android, A.I. research), Microsoft 
(LinkedIn, Teams, Skype, Flight Simulator), Zoom, Twitter (a global micro-blogging 
platform until its acquisition in October 2022). Tencent (Qzone, WeChat) and Sina (Weibo) 
dominate the market in China while ByteDance’ TikTok is popular around the globe. This list 
is far from being comprehensive. Games is a huge sector that is also quite relevant for VR 


because level designers already have the know-how to create engaging 3D worlds. 


The most important revenue stream is selling ads. Therefore the social media platforms do 
massively collect user data to offer micro-targeting services to marketeers. For short: user’s 
online time and behavioural usage profiles are sold to run targeted commercial and political 


campaigns. If you are not paying for a service, then you are the product. 


Cooling down. Back to text. 


Cool Reading 


Reading is a linear repetitive activity. It is a fast cascade of focussing the words to harvest 
their meaning. Not every word is deciphered one by one. Instead the eye jumps 3 to 7 times 
per line to send sharp signals to the brain. Frequent reading improves the ability to detect 
certain patterns in the shape of text to obtain the meaning quite efficiently. 

Reading a detective story remains linear even in case of cheating: Reading the last 
pages first is just a different order of reading the one-dimensional text. Scientific papers use 
footnotes or offer supplemental material in the appendix. Reading is optional; it’s up to the 
reader to take any way through a text. 

Even reading hypertext is a linear activity. At certain points in text-space and personal- 
time the reader makes a deliberate decision to jump to a next chunk of text. Therefore 
browsing hypertext remains personally linear. However the reader (or user) might get lost in 
cyberspace. Then it is a matter of information architecture to provide a useful and usable 
navigation structure with sufficient hints to guide the reader (or user) along an intended trail. 

According to Marshall McLuhan speech is a cold medium: »so little is given and so 
much has to be filled in by the listener.« [Marshall McLuhan, 1964: Understanding Media, 
chapter 2]. Even more so when speech is delivered as text. Intonation, mood, and any body 


language of the speaker or author are missing during a pure reading experience. 
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Reading is a cool activity — like in cool jazz. The reader has to contribute her own 
background and fantasy to unfold the whole story. Reading text stimulates the brain to create 


a mental theatre with the plot and ideas that are encapsulated in black ink on white paper. 


A similar phenomenon is called closure [Scott McCloud, 1993: Understanding Comics]. The 
reader of comic strips has to close the gaps between frames by imagining the missing 
pictures. [c.f. The New Yorker cover, Feb 25, 2008; via Barbara Tversky’s chapter in this 
volume] {While Naomi’s eye cascades over the reference, the image dissolves next to the 
paper. A gaze causes the image to zoom and she ponders a book’s shelf life. } 

As a visual 2D medium, graphical novels are still a cold medium, while movies are a hot 
medium — to follow McLuhan’s terminology. There is no need to apply imagination to 
complete the rich visuals and Dolby surround audio of blended effects and a symphonic 


music score. 


Hot VR 


Much like movies, virtual reality (VR) is a hot medium. The user experiences a 3D world 
which is projected into a sphere of pixels and an endless audio track is playing over 
headphones. Alternatives to head mounted displays should be mentioned as well: For instance 
the CAVE (Cave Automatic Virtual Environment) is a stereo projection inside a box — large 
enough for a human to make a few steps. Other systems use large rooms covered with curved 
OLED displays behind protection glass on the floor. Amusement parks try to attract people 
with 360 domes — similar to planetaritum’s night sky projections. All systems have some 
advantages and also some drawbacks for certain contexts of use. Technical requirements, 
affordability, ergonomic form factor of the hardware, availability and compatibility of 
software, interoperability with other computer platforms, interactivity, the lack of well 
established VR design patterns and poor usability... just to name a few issues that need to be 
addressed. 

On the other hand there are several features of VR that make the platform desirable and 
interesting to explore new concepts — not just for gaming. VR offers more degrees of freedom 
than TV or cinema, i.e. the user can turn the head to look around, change her position by 
“walking”, and interact with virtual objects by “touching” “buttons” and “pulling” “levers”. 
Hand tracking and gesture recognition is necessary to interact with virtual objects. 

The term immersion is used as a quality measure how convincing the VR experience is, 
whether the user believes to be “really - there - now”. The sensational impression of presence 


is supported by high resolution 3D graphics, high refresh rates, and extremely short lag times 
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on turning the head to mitigate motion sickness. 


It depend on the implementation effort of the development team whether believable creatures 
or humanoid characters populate the scenery and whether ambient sound provides subtle cues 
und realistic flavours during acting inside the VR environment. Good quality in all these 
aspects is necessary to offer an immersive experience. 

VR is a slightly cooler medium than TV because the user can interact with the scenery and 
change the flow of events. In other words, VR requires physical und mental user participation 
while a cinema experience can be watched and enjoyed quite motionless from the armchair. 
But VR is definitely a hot medium compared to text because reading text requires creative 


imagination to revive the written words. Ready-made VR world just need to be observed. 


It always poses problems when cold and hot medium categories compete on the user’s 
attention. Images draw attention over text. Videos draw attention over text and images. As a 
young medium, VR requires the most amount of lead time to get started before use. The 


perceived cost/value relation of reading in VR is just too high at the moment. 


Real Text in the Virtual World 


Text is text independent from the medium, whether it be paper or pixels. But since 
McLuhan’s »The medium is the message« we must consider the channel, the display 


properties, the interaction design, and the social context. 


Text in the post-paper modality is mostly used for news and information or for personal short 
text forms like e-mail, micro-blogging, public or private chats, and texting <sic!>. Books 
have not fully completed the transition into the digital world yet. Too rigid the software 
compared to paper — too tiresome the reading activity itself. Better display devices with 
higher resolution, higher refresh rates, or even electronic ink offer an experience of text that 
is as stable and legible as printed text on paper. However, a few issues remain: digital text is 
not spatially persistent. It always depends on the tool and the recent click or swipe activities 
how and where a paragraph is displayed. Hence it is a desperate attempt to look for a 
paragraph that was located somewhere on the upper third of a right page roughly after the 
introduction. Other interaction challenges are personal highlighting and annotations. Some 
propriety silo solutions are available. But none of them is as flexible as pencil scribbles on 
paper or as standardised and connected as the Web itself. None of the annotation solutions 


explores the realm of dynamically connecting people and media. 


Display quality gets better. Goggles get smaller and more ergonomic to wear for longer time 


periods. Lab experiments are being conducted to use contact lenses instead of clunky 
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headsets Hopefully interface capabilities and usability for reading and annotating text and for 
text authoring tools will improve as well. 

Text in the really real life — excluding the printed word and the digital domain for a moment — 
occurs in public urban spaces. Text IRL is used on highway signs, as street labels, signage on 
and in buildings, even as hints on doors — PUSH/PULL — not to bump your head. Text IRL is 
used on billboards; picture the neon marketing messages on Times Square or Piccadilly’s 
large urban displays, which blend into the digital world already. Text IRL has a purpose to 
inform the “users of RL” about certain features; for instance how to find your way in a city, 
or which coffee to order in a restaurant. Text IRL supports RL by delivering necessary or 
superfluous information to the inhabitants of the space. 

Text IRL without a function might be considered as art. There are a couple of examples for 
this category. Maybe graffiti? Maybe city branding campaigns like the letter sculptures 
Iamsterdam. Certainly urban word art which makes the pedestrians slow down and ponder 
the philosophical relation between letter sculptures and the location. 

These considerations are quite relevant for VR if you acknowledge that artificially created 
reality aims to mimic the real world until the scenery becomes indistinguishable and the 
sensational impression can be considered perfect. The real world is the primary metaphor of 
the virtual world until it passes a VR Turing Test. 

Virtual objects might stimulate our senses like their counterparts in the real world. Flipping 
through a virtual book might provide a sense of weight, haptics of paper, the sound of waving 
sheets, eventually even a fresh breeze of air or the smell of yellowed paper. A gesture with a 
finger is sufficient to flip (or scroll?) though the pages. 

Initially a new medium will embrace all content that has been created for prior generations of 
media technology until the characteristics become clear and evolve into a new medium of its 
own right. Hence it is no surprise that several Hollywood movies depict the future of VR in 
quite classical terms. Three movies stand out: In »Disclosure« [Barry Levinson, 1994; based 
on a novel by Michael Crichton, 1993] the VR user virtually walks through a virtual library 
and opens cabinets to look for specific documents in virtual folders. »Minority 

Report« [Steven Spielberg, 2002; John Underkoffler as a consultant for presumable user 
interface concepts] introduces hand gesture interaction on large curved screens to sift through 
a huge media library to find evidence. The user moves like a conductor in front of an 
orchestra to skim through image and video footage. Finally the »Matrix« trilogy in 4 parts 
[Lilly and Lana Wachowski, 1999-2021]: VR is indistinguishable from reality and the only 
perceived state of being. There are only a few glitches in the matrix that causes suspicion 
about his perceived reality for the hero Neo. {»Follow the white rabbit,« Naomi mumbles. At 


the periphery the scene from Matrix fades in. } 


420 


All of these Hollywood interaction design video prototypes are impressive — that’s part 
of the success of the movies. But do they represent a usable and desirable concept for text in 
VR as well? It is more likely that VR will be a 3D TikTok horror show with billboards, 
subtitles, speech bubbles; more like massively multiplayer online games (MMOG) with 
plenty of targeted marketing messages. 

The cold text medium and the fairly hot VR medium do not fit together. The high definition 
environment will swamp any cold text medium that appears as a shy digital object. The 
virtual world offers so many attractions that the users cannot focus her attention on longer 
text blocks to read. The same is true for writing. Too many distractions provide a poor 


environment for sound reasoning or to create engaging stories. 


A Vision for Text in the Virtual World 


Like in the real world the environment matters for concentrated reading or creative writing. If 
anything is possible in VR, then dedicated 3D rooms should be designed and offered to 
support authors and readers. Interior designers might be involved to create cozy and calm 
rooms which display the corpus of text as primary digital objects. Related material is within 
reach. Significance can be mapped to distance. Filing and retrieval of documents should not 
simply mimic real library architecture where long and narrow aisles lead to sky-high book 
shelfs. In real life shelf space is a scarce resource. Space in VR is endless. Effective and 
efficient navigation structures are crucial in VR. The visual design of VR libraries shall not 
resemble the aesthetics of sci-fi movies. Instead some imagery of real and therefor familiar 
libraries might set the mood and expectations to interact with the collections. Mood images 
work like icons and labels and provide orientation to the user. Algorithmic magic shall 
augment and assist the user’s ability on browsing papers and connecting the dots for new 
creative conclusions. Interacting with resources should not be any simpler than the motto 
»information at your fingertips.« The action to offer more material or to visualise concepts in 
animated 3D graphics must only be a response to a clearly articulated wish of the user — such 
as the tip of a finger or a mumbled command. Otherwise the focus of attention is allured to 
different media. 

A new interaction language for gestures needs to be established. We’ve had mouse 
clicks and drag’n’drop for desktop WIMP systems (windows, icons, menus, pointing device). 
Swipe, pinch and tabs are finger gestures on mobile touch devices. 

Take the full body tracking from »Minority Report«. Any gesture can be interpreted to 
control the virtual environment. Raising an eye brow, nodding the head, shrugging the 


shoulders, conducting with both arms... The possibilities and degrees of freedom to trigger 
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actions in the VR environment are tremendous. Therefore it is necessary to establish vendor- 
independent conventions how to interact and behave in VR. The systems will adapt to 
individual preferences and habits like they do today for speech recognition. A prediction 
model will always calculate the user’s intention based on the current context and be ready to 
offer related information on demand. Gentle micro feedback — visual, audible or haptic force 


feedback — tells the user about the responsive state of the system. 


Augmenting Human’s World 


Augmented reality (AR) will adopt the interaction paradigms from VR. In addition, an 
internal digital twin of the real space needs to be kept up-to-date. The AR experience might 
be more comfortable and satisfying than being in a VR world because the natural and 
therefore familiar environment is always present and can be used as a reference point and as a 
backdrop to superimpose digital text and other media. Real surfaces become interactive 
displays. Sticky notes become virtual sticky notes that can be placed on augmented surfaces 


or on virtual work spaces. 


Collaborating with other people in shared AR environments can also be a productive setting; 
less for writing text, but to inspect and create hovering models in space. 

Alan Kay shared an anecdote from times when he was a student at University of Utah in the 
mid 1960s. Alan and a class mate got the assignment to improve a Simula program. An 
endless paper printer has produced an almost endless printout of the program. They rolled out 
the paper “scroll” down a hallway. While crawling across the paper they shouted their 
findings to each other to understand the object-oriented principles of the programming 
language. (Later this experiences helped Alan Kay to shape Smalltalk) — The hallway 
scenario makes sense in VR or AR as well. An innovative approach would be to identify 
problems and scenarios (for dealing with text) that can be tackled easier in an infinite 3D 


space than with a windows environment or even on small mobile screens. 


Finally, the paper metaphor get less relevant. Typewriters are exhibited in museums. DTP 
(desktop publishing) word processing, electronic mail among other means to communication 
online are common practice for more than a generation. Reading and writing text on screen 
does not have to refer to the paper mataphor anymore. People grow up with swiping text on 
smart phones. Pupils and students are always connected on free wifi. Autocomplete is the 
preferred input method for virtual on-screen keyboards. Voice UI is used for home 
entertainment systems. Although, voice-to-text still has to been proofed as a viable input 


modality for longer texts. 
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Provisions for the Future 


Josh Clark was concerned regarding “Natural User Interfaces” for touch devices. He said, 
»We are creating the illusion that there is no user illusion anymore.« 


We — as interaction designers — are diluting ourselves when we aim towards this 
objective for VR once again. There is always a conceptual design layer and a technical layer 
between the user and the service. Any usage is alway mediated by the artificial environment. 
It is the responsibility of product & interaction designers to create solutions that meet the 
expectations and needs of the users to all regards. 

Gestalt laws and human physiognomy are universal and should not be ignored. User 
centred design for AR & VR will have to find solutions that initially look and feel familiar 
even in 3D. Copying the real world can only be a first step. In the long run interaction 
paradigms of desktop and mobile will be extended to utilise virtual 3D world that is projected 
into a 360° sphere or augmented onto the real world. Free floating windows in space is 
merely a minimum viable solution. “Physical” motion and hole body gestures will be added 
to the interaction modes of mouse, multi-touch and voice. The virtual depth of VR can be 
used to create primary working areas, secondary side spaces and rooms in the vicinity for 
other resources or other primary activities. Rooms offer a specific set of actions. Rooms can 
be considered like apps today. Multi-user environments need to pay attention to privacy 
concerns in shared spaces. But they offer the opportunity for collaborative dynamic spaces to 


tackle wicket problems collectively. 


{ Naomi moves two finger downwards followed by a thumbs up gesture. The matrix of 
previews shows stacks for each chapter of »The Future of Text«, volume 3. Some stacks look 
a little bit crumbled. She will continue with Mez Breeze’s article tomorrow. The room lights 
up again. Naomi still prefers to actually read instead of having a SmartAssistant reading it to 
her. 


mail: mprove@acm.org | mastodon: https://hci.social/web/@mprove | web: https://mprove.de 
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Mez Breeze 


Artificial Intelligence Art Generation Using Text Prompts 


With novel terminology such as ‘image synthesis’ and ‘latent spaces’ percolating rapidly 
through the AI arena, the realm of Artificial Intelligence in relation to art generation is 


currently accelerating with breakneck speed. 


Since the advent in early 2021 of OpenAI’s text-prompted image generation program DALL- 
E, an explosion of AI text-to-image generators have emerged including Midjourney, Stable 
Diffusion, Imagen, Craiyon, and NightCafe Studio. Along with this burst of AI art generators 
harnessing text in a very functional way — as text-to-image crucibles — the corresponding 
wave of image synthesis is instigating a fresh reliance on, and examination of, the role of text 
itself as an imagination engine, with accompanying microstories proliferating alongside AI- 
genned imagery. With each update and/or subsequent jump in the innovations these AI art 
generators are providing creators, there’s a corresponding surge towards text exploration and 
experimentation especially in terms of explanation, description, and narrative manufacturing. 
Alongside these surges is the need to develop associated ethical guidelines and best use 
principles when using these text-prompted AI art generators, including rules for prompt 
engineers, and the moral - and potentially legal - minefields it provokes. This paper will trace 
such explorations, experimentations, and ethical considerations associated with the use of 
using such text-dependant AI art generators, while outlining the concepts involved in text-to- 
image synthesis and the process of text prompting through an examination of the AI-human 
collaboration ‘[Por]TrAlIts: AI Characters + Their Microstories [Book One]’. 


https://mezbreeze.itch.io/portraits-volume-one 
Beginnings 


So it’s 1988 and I'm hungover and crouched over a desk half-heartedly watching my 
University lecturer give a talk about societal impacts and future trends. At one-stage the 
lecturer uses the term Cyberspace, a concept which at the time is new to me, but it gets my 
attention to the point where I'll later look up the term (when I'm less hungover) and have my 


tiny 17-year-old mind blown by what I find. 
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Jump to 1994 and I'm sitting in an offwhite computer lab where a friend, a mechanical 
engineering student, is telling me all about the wonders of the Internet and the World Wide 
Web. After she leaves I dive full force into the guts of programs like Telnet, Fetch, and 
Mosaic: and thus begins my becoming hooked on the joys and terrors of the Internet and the 
World Wide Web. 


Jump again to the year 2022 and I am sitting in my studio listening to the founder of an 
Artificial Intelligence organisation who is currently onboarding us, a group of beta testers, 
who have been invited to test their AI image synthesis generator. This is not my introduction 
to AI Art generators which happened a few years prior, but it is still a pivotal moment where 
the true societal and cultural impacts of such technology start to manifest in my limited 
consciousness. 

In all three instances just described, each encounter can be viewed as a milestone 
regarding introductions to, and interactions with, technology that would (and will) proceed to 
shape our contemporary world for better and/or worse. Cyberspace, the Internet, XR and VR, 
and Artificial Intelligence have had (and will continue to have) explosive societal impacts. In 
terms of the cultural gravitas with which they should be viewed, just as the Internet has 
become inextricably embedded into the very fabric of humanity's relationship with 
technology, Artificial Intelligence in general — and the use of text in relation to the production 
of Al-generated artwork using text prompts specifically — holds the potential to critically 


impact industries, institutions, individuals, and societies at large. 


The Stage 


Since April 2021 when OpenAI’s text-to-image generator DALL-E intro-splashed across the 
AI scene, AI Art generators have burst onto the ‘next-big-thing’ stage in spectacular fashion. 
If you’re a regular user of social media, it’s highly likely that you’ve recently (and regularly) 
been exposed to a stream of AI imagery shared by eager creators using text-to-image 
generators like DALL-E2, Stable Diffusion, Midjourney, Craiyon, NightCafe Studio and 
Imagen who are keen to explore and in some cases unfortunately exploit such methods of 
producing visual output. Creators, developers, critics, academics and commodifiers all seem 
keen to jump on the AI Art bandwagon and hitch their fortunes to the next tsunami tech- 


wave. 


The Lowdown 
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Text-to-image AI Art generators are for the most part based on neural networks trained on 
massive datasets (some of which are scraped from the Internet itself) that produce output 
through Diffusion, a practice where images are produced effectively from randomness 
through a process involving image noising and denoising. Using text structures called 
prompts that contain a combination of descriptors, tokens, styles, punctuation, modifiers and 
concepts, the resulting visual outputs can mix the strangest combination of elements to 
produce unique results. Such image synthesis occurs out of what is termed a ‘Latent Space’ or 
a type of abstract, multi-dimensional limbo which contains visual potentialities dependent on 
the datasets used in the AI training which are almost ‘summonsed up’ from this space by a 


specific combination of words and punctuation. 


In the rush to embrace text-to-image generation, the term prompt engineers - used more 
broadly in machine learning - is now being co-opted to describe people using such text-to- 
image generators who craft such prompts. Prompt engineers deploy text in particular ways to 
direct their desired image output, with manipulation and experimentation playing a key role. 
This need to play with text and semantic structuring has fostered a fast-moving subcultural 
base, one that is gestating and evolving rapidly with digital spaces like Github, Hugging 
Face, Replicate and Google Colab being harnessed as playgrounds in which to test such 
experiments. Alongside the role that text takes in such explorations (that being one of an 
imagination engine), with each AI Art generator upgrade or modification like inpainting, 
outpainting and upscaling, the AI Art field becomes broader and more adventurous. Text-to- 
image adopters are also using text in novel ways alongside their generated AI Art, with a 
surge of poetic and fiction-based stories proliferating — AI Artists like Vladimir Alexeev and 
Dr Siobhan O’ flynn have been using such AI output as the backbone of larger story-based 
projects. 

One such storytelling project I’ve been constructing since July this year is a book series 
based around text-to-image output. The first book in this series, [Por]TrAIts: AI Characters + 
Their Microstories [Book 1] is a collaborative effort between myself and the AI DALL-E2. 
The book is comprised of a fusion of AI generated portraits and microstories written in my 
signature English/code-hybrid language called Mezangelle, and was inspired by being invited 
by the OpenAI team to participate in their AI Artist Access Program (OpenAI being the 
organisation responsible for creating DALL-E2). This book can be accessed here: https:// 


mezbreeze.itch.io/portraits-volume-one. 


The Impact[s] 


Academics, artists, and non-artists of all stripes have begun to wade into debates concerning 
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the legitimacy of text-to-image generated art, with a large percentage of the resulting 
dialogues veering predictably towards the hyperbolic. There are also valid concerns being 
expressed by certain societal sectors regarding the potential seismic cultural shifts that might 
well be associated with such AI tech, and although the term ‘disruptive’ has been ridiculously 
co-opted to represent anything even vaguely associated with non-standard use, I can’t really 
think of a more apt arena for true disruptive impact than AI text-to-image (and very soon 
text-to-video, text-to-animation, and text-to-game) generation. [Author Note: examples of 
how quickly AI generative fields are accelerating can be seen in the fact that in the three 
weeks since this paper was presented at the 2022 Future of Text Symposium, Google and 
Meta have both released new text-to-video generators and Microsoft has announced the 


inclusion of text-to-image AI Art generator into a new Office app called Microsoft Designer. ] 


Just some of the creative industries and individuals likely to be heavily impacted by the 
growing use of AI Art generators include graphic design, concept artists, photography, 
illustration (storyboarders, cartoonists, plastic arts practitioners), film and video editors, 
curators, animators, game developers, and interactive storytellers. And this doesn’t even 
cover how AI will impact and is in fact right now affecting businesses like stock photo 
outlets, advertisers, and publishers. In fact, there’s already been several controversies 
surrounding the use of text-to-image generators, such as the furore over the Midjourney- 
crafted AI image that recently won a Colorado State Fair Digital Artwork Prize, and the 
disturbing report of real medical photos being included in datasets used to train AI image 
synthesis models. There’s also the fact that as of September 2022, Getty Images have banned 
any AI works from being uploaded or sold via their platforms. 

It's been extremely enlightening being involved in the both the DALL-E2 and Stable 
Diffusion Beta Testing programs: alongside the absolute wonder and delight in using such 
tech, seeing inbuilt implicit bias concerns arise has been less than ideal, as has problematic 
content leaning towards misogyny, hatespeech and racial stereotyping, as well as training sets 
(and even worse, beta testers themselves) perpetuating the myth of the 'great' male artist with 
some women-identifying artists and representations being relegated to muse status and/or 
male gaze fodder. But this is just the tip of the AI Art generation iceberg: other issues being 
raised over text-to-image art include legal ramifications regarding copyright, the ease of 
propaganda creation, and problems surrounding the datasets on which generators have been 


trained that include living artists work without any permissions or compensation given. 


The Rules 


The development of crucial ethical guidelines and best use principles when using text- 
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prompted AI Art generators, including rules for prompt engineers, can’t come soon enough. 


Some ethical questions that might be considered when using such text-to-image generators 


include: 

1. When writing text prompts, do you absolutely need to include references to particular 
artists, living or deceased? 

2. If you must include artists in prompts, should you make sure to use a mix of many 
artists and styles, and preferably only included the names of deceased artists? 

3. Think long and hard about whether it is acceptable to use text-to-prompt generators for 
the creation or dissemination of hateful or harmful content. 

4. Consider if your text prompts replicate or emulate any overarching biases or lopsided 
power structures, and if they do, whether it is ok to a) use them in the creation of AI Art and 
b) to promote or publicise such images? 

5. Consider the implications of the long-term use of AI Art generators on our increasing 


Climate Emergency due to image synthesis being extremely computationally demanding. 


Conclusions 


Jumping back to that hungover morning in 1988, if I’d known then what I do now about the 
wide-ranging impact of Cyberspace and the Internet in general I’d probably have laughed and 
then sobbed, especially given the epistemic crisis humanity is currently facing due in part to 
the development and use of such technologies. Just as in the 1990’s I would never have been 
able to predict how impactful the Internet would become, it’s almost impossible to ascertain 
just how text-to-image AI Art generation will manifest in the future — but if we’ve learnt 
anything from the past (including the weaponisation/politicisation of text and media in 
relation to propaganda and political grandstanding) we’ll be wise to hardbake lessons learnt 
from such technologies into all aspects of AI Generation, including the textual nuances and 


moral considerations relevant in the construction of such imagery. 
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Michael Roberts 


Metaverse Combinators: digital tool strategies for the 2020's and 


beyond 
Introduction 


For the last 30 or so years, one of the dominant paradigms in tools for making digital content 
has been the “node-code”, “flow-based programming” or “visual programming” style more 
accurately referred to as “node-based programming”. 

In these tools, digital content is expressed by connecting together “nodes” into graphs 
using edges or “wires”, with this variant of programming being commonly known as 
“wiring”. 

Modern examples of the genre include Touch Designer™, the Maya Hypergraph* Epp 
Blueprints’ and others. 

Example applications of this technology have ranged from shader editing all the way 
though to controlling high-level behavioral interactions, as would normally be performed 
using some sort of conventional scripting language, such as Lua or JavaScript. Application 
areas are now extending to distributed network applications. 

The tools typically exhibit much finer grain control of the “look and feel” of digital 
content than the current direction of using natural language input to “generate” Al-based art, 
which is disrupting the art content generation pipelines. They allow users to “tweak” on 
minor visual appearance properties in a way that is not currently possible with text input AI 
tools. 

On the other hand, in AI tools “complexity is free”, meaning that users to not have to 
explicitly code or “make” it. This paper attempts to analyze some of the potential 
interactions between these two, non-mutually exclusive, paradigms, as well as provide some 


“language” for discussing such relationships. 
Programming using node-based languages 


Standing in contrast to conventional textual programming, node-based programming straddle 


the gap between so-called “real programming” using conventional textual languages (typed 
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or untyped) and simple, typically database driven, configurations. 


Node-based languages allow the user to use or combine small modules (or nodes) 
which express limited behaviors controlled by parameters (or properties). The behaviors of 
the concert system expressed by such tools thus controlled by 3 elements: the selection of the 


nodes used; how the nodes are connected together (topology) and the properties of the nodes. 


Nodes typically express “ports” which are used to connect to other nodes. The act of 
“wiring” involves drag-and-drop type operations in which “wires” or “edges” are connected 
between the ports as well as the setting of properties for nodes using some sort of property 
editor. 

In a 3D/VR, multimedia or a similar context, nodes can function as “media objects” — 
with high level properties such as “resource locators” that point the runtime system to load a 
mesh, character, sound, area of text, or other media object. Likewise, lower-level surface 
appearance data, such as mesh textures, procedural geometry, height maps, etc., can all be 
expressed by graphs with appropriate runtime and node support. 

A variety of underlying implementation mechanisms have been used as execution 
engines for such graphs. In the author’s current work, this engine is a parallel message 


passing virtual multicomputer, but other approaches have included compilation into machine 


code or intermediate representations such as SPIR-V*®, conventional sequential textual 
languages, dataflow models, execution using function evaluation or, in the author’s 1989 
work (Roberts & Samwell, 1989) (Roberts, 1990), compilation into a parallel programming 


language. 
Combinatorial thinking 


Combinators are a higher-order functions that uses only function application and previously 
defined combinators to derive a result from supplied arguments. 

Based on original work by Moses Schoénfinkel and Haskell Curry in the early 20’s, 
combinators have found widespread application in functional programming though languages 
like Haskell*?. 

The essential idea behind combinators is that function state is bound into the function 
invocation using only bound variables (as opposed to free variables) — I.E., arguments to 
functions, and then functions are combined to end-results of arbitrary depth. 

However, “constants”, such as “1” in a function declaration, can also be considered 
“state” when viewed though a certain lens (changing the function by changing the 


“constants”). Such splitting of hairs leads to a grey area between object-oriented and 
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functional languages in which the act of editing a function definition can be regarded as 
“changing the state of an object” — analogous to the setting of properties on a node in a node- 


based language. 


Similarly, the act of connecting nodes together with edges can also be regarded as 
combination — or the process of enumerating combinations or configurations of the smaller, 
simpler functions “contained” in the nodes. We therefore use the term “combination” and use 
this to refer to the act of programming a node-based system via the connecting of various 
nodes together into a working system via wires and the setting of their properties. 

Node-based languages are deeply combinatorial, as are other systems commonly in use 
for digital art, like painting programs. Such programs define a “combinatorial space” which 
can be explored by users making digital artifacts — artists serve as navigators of such a space, 
making aesthetic choices and exploring pathways though the space defined by the tool that 
make sense both from a cultural perspective and also with their own sense of how things 
“should be”. 


Meta tools 


Metaprogramming is generally held to be a programming technique in which computer 
programs have the ability to treat other programs as data. It is a part of the genre of thinking 


which believes that tools should have the ability to make more, and higher-level, tools. 


Traditional crafts have a notion of a “mother craft” or “fertile” tools. These are toolsets 
and processes which can give rise to artifacts that can be used for the same or other 
applications. Blacksmithing (the traditional craft concerned with forging metal) is one 
example. Using a relatively simple set of underlying or “bootstrap” tools (hammer, tongs, 
forge and anvil), blacksmiths can forge all the tools they need to make both tools for their 
own use and also tools for other domains, such as farming, pottery, or even sewing. Over 
their lifetime in the craft, blacksmiths typically accumulate large numbers of self-forged 
tools, ranging in application from simple to complex. The entire western industrial 
revolution can be considered as emerging from this historical activity. 

The h-graph or “hierarchical” graph model used in some node-based languages holds 
one key to the development of such “fertile” tools in the visual domain. In this 
representation, nodes and edges can conceptually be “wrapped up” into a higher-level object 
— itself a “node” — the composite behaves like a “primitive node” but is itself a compound 
object formed from the combination of multiple lower-level objects, in the same sense that 


we build modern software using libraries. When such an object can, via component objects, 
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make other objects, access/set their properties, and connect them in various different 
topologies, we have a true “metaprogramming” tool, in which we can write tools “in the tool” 
that themselves make things. The result is capable of spawning complexity and up 


leveling the functionality of the toolset — a “strange loop” in the sense of Hofstadter. 


Making tools expands the combinatorial space defined by the original tool, leading to 


an entity that “grows” with time. 


Information Hiding 


One of the key critiques of visual programming languages is that they “lead to mess” — 
detractors see a mass of “boxes connected with lines”, and it is true that many 
implementations of the paradigm do suffer from this fault. Hence, visual programming 
applications that don’t implement “information hiding” force users to consider “all the nodes” 
at the same time — an overwhelming process give the large amount of bandwidth available in 
our visual system. 

We think that the unreasonable effectiveness of textual programming is somewhat 
caused by the information hiding properties inherent in text — instead of navigating a large, 
complex structure “all at the same time”, good programmers wrap up their code into 
hierarchical pieces — classes, functions, and methods - which perform simple, well-defined 
operations that can be tested and reasoned about separately. The motivation for this is 
probably related to the fact that we can only hold a relatively small number of concepts 


simultaneously (normally equated to 7 (Schenkman, 2009)). 


Given that the textual representation is “opaque”, textual programmers probably mainly 
navigate a mental model, informed by their reading of the text. To be effective, visual 
programming systems must implement explicit “information hiding” mechanisms that allow 
users to flexibly consider sub-pieces of their program rather than the “whole”, which is 


outside comprehension parameters. 


Hyperparameters 


It is well known that adjacency matrices can be used to express graph structures. Such matrix 
representations (with weights) are heavily used in deep learning systems, for example tensor- 


flow>?. 


Similarly, individual low-level parameters, perhaps the properties in the nodes in the 


aforementioned systems can be encoded as “weights” in a much larger matrix structure 
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representing the overall parameter space defined by the visual programming system. The size 
and scale of these structures means that they are not readily comprehensible by humans 
because many of the information hiding properties from the previous section simple do not 


exist over this representation. 


Likewise, the scale of such a system can be heavily affected by the combinatorial nature of 
the node-based language, as well as additional parameters defining sum-of-linear function 
structures that appropriate potentially non-linear parameters inherent in the graph model. We 
have thus come to a situation in which it is possible to encode the “program” in a visual or 
other content creation tool using a matrix representation which is almost completely opaque 


to a human overseer. 


This is, in fact, exactly what tools like Midj ourney” ! and Dali-E-2>7 do, but they 
approach it from the point of view of learning the structure (and thus hyperparameters) for 


such a representation though consideration of the output from such tools. 


Once trained, such a structure cannot “grow” unless it is retrained on different input 
data, a limitation that reinforcement learning sidesteps, by constantly retraining itself by 


“playing a game and observing the results” with the domain it is working in. 
Machine learning approaches 


Midjourney and Dali-E-2 create realistic images and “art” from descriptions phrased in 
natural language which are used to activate particular sets of hyperparameters inside a 


learned representation. As such, they form part of the “future of text”. 


In a short time period, these tools have become so prevalent in popular culture that we will 

skip over a more in-depth description of the process of using such tools, and merely point out 

several key take-aways. 

¢ Such tools leverage the computer graphics tool set built over the preceding 30 years, 
because they process (as input) imagery primarily generated using such tools. The process 
of processing such images encodes properties of the images into a set of hyperparameters 
expressed in a neural network. 

¢ Likewise, they leverage and implicitly encode the individual process and techniques of 
artists who originally used the digital tools. 

¢ Techniques for texturing, painting and other mechanisms, accessed by the artists, have 
explored a significant portion of the combinatorial space made possible by the original 
tools and thus the space over which the learning operate. 


¢ AI cannot, at least conventionally, invent “new” space — it merely remixes and combines 
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hyperparameters extracted from the combined original work of the artists. 


¢ Fine details — what would normally be referred to as “technique” in art or programmed for 
in a tool (such as surface appearance) is essentially “free” in such a model — users of the 
tool no longer have to make this fine level of detail. 

* Operating at this level, imagination is “constrained” by the textual navigation method. 
Consider that I imagine a cat — if I am drawing a cat, I am going to have a lot of latitude in 
how I represent the cat. Some of this complexity (the sum of all possible input cat art) is 
expressed in the hyperparameter space of the AI model, but we currently lack the tools to 
navigate it, view it in any sort of totality, or really understand it’s nature. 

* Much of the conventional computer graphics pipeline, as exhibited by generations of 
SIGGRAPH papers, has focused on hand-encoded techniques for producing particular 
visual appearances. This pipeline, which encodes a lot of knowledge about process and 
performance, could potentially be obsoleted, if we move to a world in which “rendering 
engine output” is directly encoded neutrally, as pointed to by techniques like NERF 
(Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, Ng, 2020). 


Moving forwards together 


It is tempting to look at the “AI art tsunami” and think that the sky is falling for conventional 


tools. 


However, some companies making tools, such as Adobe, are beginning to release products in 


which AI is used to augment more conventional digital content creation tools. 


If we choose to continue to represent content in ways that make sense from a cognition 
perspective for human beings, then these representations look a lot like the tools and 
processes that have gotten us to the point we are at now and which generations of people 


have thought about with a view to simplicity of representation. 


Instead of making AI tools which make remixed content from parsing the output from the 
conventional tools, why not begin to focus on making tools which learn into the common 
computer graphics representational stack and thus unlock the combinatorial power of human 
creativity? 

For example, rather than making a tool which synthesizes images directly, rather make 
tools which generate 3D models and surface descriptions suitable for use in conventional CG 
pipelines, surfacing control over such objects in the form of the node-based programming 
which countless technical artists are already familiar with. Learning into such a 


representation also allows artists to “tweak” at the fine grain knobs and dials to obtain exactly 
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the results they want, rather that accepting art “made” to quite a vague specification. 


Such an effort clearly is not without difficulties — for example learning over complex 
parameter spaces of non-linear functions is a current open research problem. However, the 
payback is that successive generation of tools writers both learn fundamental mechanics in 


the operation of their tool and have access to AI functionality that makes life easier. 


The alternative, unfortunately, holds possibilities of a world in which the creative 
process that defines much of what we are as humans is progressively decoupled into the 


computer’s domain. 


Conclusion 


We have tried to outline with a broad brush the productive area of synthesis between 
conventional node-based art tools, and the newer digital tools based on machine learning - 
both discussing the parameterized space over which all such tools work and drawing some 
conclusions about how to think about this space. Finally, we have offered some suggestions 


for ongoing work in this space. 
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Omar Rizwan 


Journal : Against ‘text’ 


Omar Rizwan 
@rsnous 


something weird about how old-school 
Xanadu/hypertext concepts seem to dramatically 
/privilege/ 'plain text', as this well-understood basic 
layer that all your links and transclusions sit on top of 
and interconnect 


6:46 PM - Aug 31, 2020 - Twitter Web App 


Figure 1. https://twitter.com/rsnous/status/1 300565745 147863040. Rizwan, 2022. 


I don’t know if text has a future, or even if it should have a future. 

I guess, fundamentally, I’m uncomfortable with the whole framing of ‘text’. I think that 
it comes with a lot of unhelpful baggage and connotations. When I start with ‘text’ as my 
basic concept, at some level, I’m starting with English prose, and alphabetic letters, and 
keyboards, and a rectangular screen or a piece of paper on a desk, and ‘plain text’ files, 

Yes, you can say that 'text' also includes mathematical notation, or YouTube videos, or 
comics, or other writing systems, or any other media that humans have come up with, but I 
think that’s a sort of slippage. I think that if you articulate your goals in terms of text, you 
may pay lip service to all of those other forms, but you will always tend to treat them as 
exceptions and deviations from the norm. The picture in your mind will always start with the 


blank Word document or text file where you type some words in, and then you'll jam in some 


carve-outs to ‘embed’ everything else among the words>*, Things other than words will 
always be second-class. 

My background is in computing, and in programming, and in trying to come up with 
new ways to interact with computers, and I think that computing has suffered very deeply 
from the centrality of text. Maybe that centrality was understandable, say, fifty years ago— 
computers were slow>>, and text is relatively easy to store and process, after all. But today, 


our computers are more than capable of processing graphics and video and sound and other 
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rich media, and I’m struck by how weak our tools still are when it comes to anything that 


isn’t text>°. 


Omar Rizwan 
@rsnous 


gonna make a 'markup language’ where manipulating 
images and video in the document is as easy as 
typing, but if you want to add any text, you need to 
put it in a separate .txt file and add a tag to embed it 


Gy) Omar Rizwan @rsnous - Jan 18, 2021 
edit text: edit the text 


add picture: take the picture, drag it into some folder of site, add the new file 
to version control, open markup file & type in some image tag syntax with the 
path to the picture, refresh page multiple times and tweak until it looks right 


Show this thread 
7:02 PM - Jan 18, 2021 - Twitter for iPhone 


Figure 2. https://twitter.com/rsnous/status/1351319206692868097. Rizwan, 2022. 


I’m struck by the fact that if I write a paper with LaTeX, or make a Web page with 


Markdown, it’s trivial to add prose, and it’s a monstrous inconvenience to add a figure. The 


figures are the important part!> 7 Text exerts this gravity, because it’s the container, it’s the 
norm. The text lives directly in the file you’re editing (and the figures live in separate ‘mage 
files’ outside it). You’re constantly (subconsciously) pushed to explain things with text, 


because it’s so much easier at a micro-interaction level to edit text than to add or change a 
‘figure”> 

(I think that this constant low-level push to use text is a way in which computing is a 
regression from paper—on a computer, it’s so easy> * to produce and edit text that it 


dominates other®, richer, potentially more appropriate media. On a piece of paper, if you 
want to draw something in the middle of your prose, you can just draw it. Imagine if making 
these were as easy as typing:) 
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Figure 4. https://twitter.com/Sonja_Drimmer/status/1368966157106114561. Rizwan, 2022. 


(On a piece of paper, drawing is no different from writing; it doesn’t represent a change of 
mode; you don’t have to build up the emotional energy to move off your keyboard and open a 
different file and a different application.) 

Even when I’m programming—there are so many things that deserve a graphical 
representation. I see it even when IJ have a bug or when I just want to know what’s going on 
with my program. It’s easy to log text, but it’s also so limited. What if I have a pile of data 
and I want a chart of it, not just summary statistics or random samples? What if I’m working 
in a domain (like designing a user interface, or drawing a map, or designing a building) that is 


inherently spatial and graphical? Yes, I can make a computer program that produces graphics, 


but it often feels®! like ten times the effort? of producing text. Text is the default, and it’s a 
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bad default. 

As you think about the future of media, I want to make the case that micro- 
interactions will dominate over conceptual models and data structures. I think that how it 
feels is a lot more important than what the concepts are. I think that people will gravitate 


toward interactions that feel® good and interactions that are immediately at hand. 


Omar Rizwan 
@rsnous 


Replying to @rsnous and @codexeditor 


like, it should feel light and natural. it should feel like 
picking up a crayon that's on your desk! it shouldn't 
be, like, going into the Tools menu and then into the 
Annotations submenu and selecting Make New 
Annotation 


4:10 AM - Nov 15, 2020 - Twitter Web App 


Figure 5. https://twitter.com/rsnous/status/1327901730235793411. Rizwan, 2022. 


That’s why I’m so concerned with whether I have to go into a separate file, and whether I 
have to switch from the keyboard to something else, and whether I can just call a print() 
function versus having to look up some graphics library, and with what things I have to go 
out and ‘embed’ into my document as opposed to entering in place. I believe that these little 
frictions and barriers are overwhelmingly important. 


I think that we live in a world that is dominated by systems that get the micro- 
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interactions right. The iPhone, video games””, social media (scrolling®’ as a formative 


interaction®®). - 

And I think that a lot of the power of ‘text’ on the computer is that it has some really 
great” interactions associated’ with it (typing, selection, copy and paste, Unix tools, text 
editors, files...). Text has this manipulability and ‘open space’ nature’!, a bit like the nature 
of files or of objects in the physical world. There are all these operations you can do (and 
know how to do) to text. Part of this is built-up capital that already exists: the hardware 
capital that every computer has a keyboard, and the human capital that everyone knows how 


to use that keyboard. How can we get those kinds of interactions, that at-hand-ness, for other 
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media? 
But that’s also why I don’t know if text has a future. What if the smartphone is the real 


personal computer in the end’>? Then we have a future where the microphone and camera 


and multitouch surface, not text input, increasingly become the favored modes of interaction. 


Omar Rizwan 
@rsnous 


was thinking that one thing that is cool about phones 
is that they don't have hardware keyboards, so they 
don't privilege text entry, so formats coming out of 
phone world (Stories, TikTok, etc) don't take it as 
given that the primary stream of a 'document' should 
be text 


@® Omar Rizwan @rsnous - Jan 18, 2021 
Replying to @tobyshooters 


yeah i think you're right that there are surprisingly deep structural reasons for 
this -- 'plain text' as a shared format for editors, also the hardware keyboard 
which makes text editing operations really quick and natural 


10:55 PM - Jan 18, 2021 - Twitter Web App 


Figure 6. https://twitter.com/rsnous/status/135137781876923 1875. Rizwan, 2022. 


As much as anyone, I admire Douglas Engelbart, Ted Nelson, and all their colleagues and 
heirs. But I also think that there is a certain arrogance to saying that the task ahead is simply 
to complete and execute their vision, that any problems are just problems of implementation. 
What can we learn from how the computer has actually been adopted +? What can we learn 
from the actual interactions and applications that have appealed to people? What can we learn 
from the genuinely new media that have popped up on laptop screens and smartphones, that 
could not have existed before the Internet or the phone camera? 
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Omar Rizwan 
@rsnous 


"| feel like he's totally contorted his message to cater 
to 70s and 80s computer people, talking about all 
these data structures and stuff when everyone today 
just wants to know how the UI will work" 


Yeah | mean, you can even do all that architectural and 
data structure stuff they talk about and go beyond URL 
links, | just think you shouldn't /ead with it when you're 
telling people about Xanadu, because no one cares! 


it's ironic that Ted Nelson is so proud of not being a 
computer/programming person because | feel like he's 
totally contorted his message to cater to 70s and 80s 
computer people, talking about all these data structures 
and stuff when everyone today just wants to know how 
the UI will work 


1:01 PM - Dec 14, 2018 - Twitter Web Client 
Figure 7. https://twitter.com/rsnous/status/1073639143878492161. Rizwan, 2022. 


Text is a strangely (historically and culturally) specific bundle of technology to orient a 
vision of the future around. Text is important, but it’s gotten a lot of attention already. There’s 
something that’s always a little exclusionary about text. It excludes the complexity that can 
go into full-fledged speech and writing’ . It excludes inline graphics and diagrams and 
notations that are often vital tools for understanding and problem-solving. I hope that the 
future of media will be broader than that. 

And — above all — to build that future of media, I believe that we'll have to find a set of 


interactions that really work, not just a set of concepts. 
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Patrick Lichty 


Architectures of the Latent Space 


Since 2020, I have been working on elements of writing with various Machine Learning 
platforms, and these are some rough working notes of that epistemological arc, focusing on 
my work with Prompt-based image generators. Initially, I had created a project called 
“Personal Taxonomies,” in which I was painting “Asemic” calligraphies based on Japanese, 
Mongolian, and Persian calligraphies, which were fed into a GAN on the playform.io 
platform. The goal was to see if, given a large set of images, a form of “Digital Rorschach” 
by looking at the commonalities between all the images, based on Noam Chomsky’s notions 
of Deep Structures. If I fed a comparative machine learning engine based on finding patterns, 
could I find internal consistencies in my own cognitive/creative processes? I invite the reader 
to find my writings on this subject. For our purposes, this text is based on the author’s next 
step: visual concretism in prompt-based machine learning image generators, and the 
centrality of writing in the creation methods, and my aims for finding alterior spaces in 


Machine Learning’s latent image spaces as forms of concretized writing. 
Context 


Since the beginning of 2022, when I started using NightCafé, I became very interested in the 
notion of text-prompted machine learning image generation. 

The first foray into this was a visual poem, "The Martin: for Negin," which showed at 
last year's Electronic Literature Organization conference. I'm not afraid to say that it was a 
concrete animation of a poem I wrote for my wife and partner, Negin Ehtesabian, during our 
first time together in Baku, Azerbaijan, in 2018. I fed the poem into that particular engine, 


and with my voiceover and guitar improvisation, I created this specific work. 
https://www.youtube.com/watch?v=w__ O5luxZ1Q 


Because of its visual qualities, I found Night Café tedious in a short time. Next, I started 
working with Midjourney AI, which is probably one of the more middle-aged text-based AI 
programs, which now, I believe, is in its third version. It's a pay system with a well- 


established community and a wide set of tools to explore prompting. 
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Content 


NOW, HERE IS WHERE I'M BEING A TERRIBLE WRITER — GETTING TO THE 
THESIS HERE, and maybe that's part of the point — working through visual narrative in AI is 
an indirect proposition. In talking with people like Ben Grosser, Marco Cadioli, Casey Reas, 
and Talan Memmott, there are several points that I would like to make about this form of 


imaging. 


1: This form of imaging is not about art making but writing. Prompt-based AI image 
generation is a concretization of syntax in the form of the prompt that the translator decodes. 
The differentiator then maps this interpretation to the latent space of however many billion or 
so images in the database. To be more precise, these practices are about exploring the latent 
image space through text as a form of index apparatus. As discussed later, that “index” can be 


a wide range of content. 


2: Machine learning-based image generation is undoubtedly a disruptive technology within 
the creative field, and its effects have unfolded in real time. Therefore, I will not be very 


prescriptive about it being art or not beyond my ideas on writing. 


3: My colleagues and I generally think that most of the work is derivative, often looking like 
something from an old Del Ray science-fiction novel book covers or photos shot through a 


small Funhouse mirror. In short, most of it is pretty terrible, leading to the next idea. 


4: I find the prompt-based image generation process manipulative or at least scopophilic in 
nature. I find it manipulative when one types a prompt with certain flags, etc., giving them 
something back that resembles the subject entered, thus making them somehow feel 
intelligent for “controlling” an indeterminately large post-photographic AI apparatus to do 
something they want. The result is a subject that the user more or less "expects ,” therein 
being the manipulation. The prompt-response loop leads me to scopophilia in that the result 


gives a minor rush of visual excitement, pushing the user to go back and refine their prose. 


5: From talking with Ben Grosser, this is where it gets strange. Considering cybernetics and 
human-computer evolution, the human being would challenge the machine to improve and 


therefore have the human being strive to improve. This is Douglas Engelbart's notion of the 
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Augment or human-computer co-evolution. But this is not what is happening. Quite the 


opposite. 


6: With prompt-based machine learning image generation, the paradigm is flipped. The 
algorithm is training the human to adapt itself so that the algorithm can give the human 
something that it finds more acceptable, pleasurable, and so on. In short, we have computer— 
human evolution, in which neoliberal technocratic systems explicitly program, inscribe, and 


evolve their aesthetics and poetics onto the user. 


7: Let's face it; artists like to break things. I have been trying to do something with my 
machine learning work over the last six months: 

7a: I have been trying to find unusual prompts that give highly unexpected results and gently 
move them into place. An example of this is my cyber/steampunk, biomorphic assault tank 
with big, fuzzy cat ears. Aleatorism to provoke surprising results is equivalent to trying to see 
what's behind the curtain. One does this using a tightly constrained set of prompts to explore 
usually unseen quadrants of the latent space. 

7b: This is used in tandem with the machine learning system’s adapting itself to the user's set 
of prompts so that the modulation of my text is dynamic in coordination with the feedback 
received from the machine learning engine. This subtlety is fascinating. 

7c: With my deep ties to the Fluxus movement, I'm also trying to see what element of the 
improvisational is in this process. Prompt as Fluxus score. 

7d: With all due respect to my colleagues for the following language, I try to break the 


machine, to see the cracks in its sense of logic, or generally get chaotic. That's what artists do. 


8: To compound this, I have been feeding the resulting series into other artificial intelligence 
algorithms. Various time stretches in Adobe Premiere taken to extremes to create other 


artifacts within my "texts." 


https://www.youtube.com/watch?v=-t2IEFECQLg&t=46s 


9: I do all this generally with the following constraints: no people, no animals, no landscapes, 
no architecture, and a few other nonrepresentational terms in my prompt set. Working in 
these tight, non-representational is an attempt to get into the weird little corners of the latent 


space. 
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I've been trying to find the places where others are “Not”, and I've made 14 series based on a 
single epistemological arc spanning about 700 images, with about six or seven other sets of 
pictures that are divergent but still try to seek the outré in the latent space. 

Other strange things underway are taking sections of my whole genome sequence, 
which I have a digital copy of, and throwing chunks of amino acid sequences into the engine, 
which has yielded exciting results. Another one was when I was lecturing on Marshall 
McLuhan in my media history class; I fed a number of his thought-probes into Midjourney AI 
and was not entirely surprised to see the images lining up very closely with the images. 
McLuhan himself might have found that fascinating. 

Also, in line with McLuhan, I am highly fascinated and suspicious of this technology. 
In the spirit of his "do you mean that my whole fallacy is wrong?" axiom, I submit this rather 
lengthy musing on the subject. It will eventually become a paper explicating that I have been 
obsessing over these processes. But, in no way do I believe in Machine Learning images 
having any veracity in themselves. 

I hope this little piece of thought lands favorably, as I have been ruminating through the 
start for a few months but have committed very little of it to text. Thanks to Scott Rettberg 
for urging me to send this across. 

If you are interested in some images, look at my Facebook and @patlichty_art feeds on 


Instagram. I've shown only about 8 out of 800 so far. 


https://www.youtube.com/watch?v=-t2IEFECQLg&t=160s 
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Phil Gooch 


Guest Product Presentation : Scholarcy 
https://youtu. be/pdVHOoh-EL8?t=899 


My name is Phil Gooch, I founded a company called Scholarcy about three years ago. 
But my interest in interactive text goes a bit further than that, mainly through the field of 
natural language processing, which is what I did my PhD in. And really, I was trying to solve 
this problem that I had when I was doing my PhD, was that, discovering new materials to 
read wasn’t a problem. I discovered lots of papers, lots of resources, I downloaded folders 
full of PDFs, as I’m sure you all have sitting on your hard drives at home, and in Google 
Drive, and in the cloud elsewhere. So I had all these documents I knew I needed to read, 
and I wanted to try to find a way of speeding up that process. At the time, not necessarily 
connecting them together or visualising them, but just really pulling out the key information 
and just bringing that to the forefront. So I started building some software that could try 
and do this. And what emerged was something called Scholarcy Library, which I will show 
you now. 

Scholarcy Library is like a document management system. You upload your 
documents, and they can be in any format. They can be PDFs. They can also be Word 
documents, they can be PowerPoint presentations, they can be web pages, they can be LaTeX 
documents, they can be pretty much any format. And what it does is, if we look at... This is 
the original PDF for one of the papers that I’ve got in my system. So this is a typical PDF, in 
this case, it’s like the original author manuscript that’s been made available ahead of time, as 
the open-access version of this paper, if you like. And as you’re obviously familiar with PDFs 
that aren’t created in software such as Liquid, Author that Frode has built, most PDFs don’t 
give you any interactivity or anything at all. You can’t click on these citations or go anywhere 
from here, for example. So the first thing I wanted to solve was if I bring this into Scholarcy, 
what does this look like? The same paper. Well, the first thing it does, is try to pull out what is 
the main finding of the study and it brings that to the forefront. The other thing that it tries to 
do is take the full text and then make citations clickable. So you saw in the original text, the 
original PDF, that citations aren’t clickable. So my first goal was to make citations clickable. 
So I can go on to that citation and go straight to that paper, and then read that, or pull it into 


my system and link it together. 
So that was the first goal. And then, the second goal was, well, once I’ve broken this PDF 
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down, can I do things like extract the figures and so on, and again, just make them first-class 
citizens so I can zoom in on them? And that was the other goal. And this all really then turn 
into a process to turn documents into structured data. And so, what I’ve built was this back- 
end API, which is freely available to anybody. It’s in the public domain. It’s not open-source 
code, but it’s an open API, so anyone can use it. If you go to api.scholarcy.com there are a 
number of endpoints here, which look a little bit esoteric. So there is some documentation on 
this on GitHub. If you go to scholarcy.github.io.slate there’s a whole bunch of documentation 
on what this API does. But essentially, what it does is, you give it a document, such as a PDF, 
but it doesn’t have to be PDF, and you upload it, and it basically turns it into JSON. And as 
you know, once you’ve got JSON, or you’ve got XML, or any structured data, then you can 
pretty much do what you want with it. So what it’s done is turn that into JSON with all the 
information broken down into key-value pairs. So you’ve got a key for the references and 
you’ve got all the references there, you’ve got a key for the funding group that has been 
broken down. So once you’ve got structured data, then it’s quite easy to bring it into this nice 
interactive format like this, where everything is, kind of, hyperlinked and clickable. So we 
can go straight to the study subjects and so on and find out there were 16 people involved in 
the study. And then, we can deal with things like, what were the main contributions of the 
study, and we can just scroll down, click on one of those, and it takes us straight to that 
finding. So the idea was really, I suppose, not really to cheat, but to, basically, speed read this 
paper by highlighting the key findings and making them clickable, as we made the references 
clickable. So all this is in the JSON data that’s underlying all this. And then it makes it more 
interactive, basically. So that’s the goal. 

That’s all well and good, basically what we’ve done is turn PDF into an interactive HTML 
with clickable citations and expandable and collapsible sections. But obviously, you want to 
deal with more than one paper at a time. And so, what I looked at doing was building 
something that could turn this into linked data. Now, I didn’t want to build a new piece of 
software like Noda, or a triplestore, or anything like that. And so I found that the Jingua 
franca for a lot of new tools that try to connect stuff together is this format called Markdown. 
One of the tools I use for hosting Markdown, and you may be familiar with it, is called 
Obsidian. But there’s many tools like this there’s Roam Research, there’s Bear, there’s 
Logseq, there’s a whole bunch of tools, I’m sure you’re aware of, that handle Markdown 
data. So here is that same paper that was once a PDF, now in Scholarcy. But now we can 
export that to Markdown, and we can do this one at a time. Or if I really want to, I can export 
all of them in one go. So I’ve only got four here, but I could have 400. Export them all in one 
go as Markdown and then I can load those into Obsidian. Put that in my Obsidian library, and 
when I open that in Obsidian, it looks like this. So it’s the same data but now it’s in 
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Markdown format, now it’s editable so if I want to, then, edit the Markdown in here I can go 
away and start doing that and visualise it. But now we’ve got the same information along 
with all the other papers that I had in my collection that were also converted to Markdown 
using Scholarcy. I can connect them together. 

So, I’ve got some of the key concepts, if I click on one of those, then it shows me other 
papers in my collection, like this one that also talks about functional connectivity. I can go 
straight to that paper, and then, I can see the main finding of that study, and I can see other 
things that it talks about, like the medial prefrontal cortex, and I can see other papers that talk 
about that. As you can see we’ve got this network graph going on, but it’s, kind of, embedded 
in the text, I can read the papers I wouldn’t have before, and I can view all the figures and 
zoom in on them. But they’re in this Markdown format where you get all this linking for free, 
which is great. And in common with other tools that handle Markdown, you can do these 
visualisations, so it’s going to show me here, if I could click on this graph view, how these 
papers are connected, in this case, by citations that they have in common. But if I really want 
to see all the concepts that I have in common, then I can click the tags view, and then 
suddenly you’ve got all these green nodes here that show me where all these papers are and 
how they’re connected by their key concepts. And as you know, Mark and I were discussing, 
just before this session started, the issue that you’ ve got too much information. This becomes 


a question of: What do we do with these kinds of visualisations? 


And I’m sure many of you here will have suggestions and ideas about how to deal with this 
because once you’ve got more than 10 or 20 papers, these kinds of visualisations become a 
bit intractable, but Obsidian lets you do that kind of graph analysis, can write queries and so 
on. Which I haven’t really done much with, my background is not, at all, in visualisation, it’s 
in NLP and converting documents from one format to another. That’s been my motivation in 
building this, is being able to turn documents from PDF into Markdown or other formats. 
And that’s what Scholarcy does. It’s a document conversion software, it’s a summarisation 
software, it gives you the key highlights of the paper, its objectives, methods, and results. It 
pulls all that automatically from each document using NLP and deep learning and it makes it 
interactive. But it does also extract into these various different formats. And one other thing it 
tries to do is to show you how the paper relates to what’s gone on before. 

So, when an author has talked about how their work sits within the wider field of research, 
Scholarcy tries to pull out that information and highlight it so it shows you where are some 
potential differences with previous work, and who’s talked about that, in terms of, 
counterpointing with what this author is saying, and again, we can click on that and go 
straight to that paper if we want to. Or which studies does it build on, for example. Does it 


build on a study by these guys? How is it different? So it’s pulling out all those citation 
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contexts, and then classifying them in, is it confirming, is it contrasting, is it just building on 
it, how does it relate? It doesn’t always get it right, but most time it gets in the right ballpark 
and it just gives you that extra context. And again, just making that information interactive. I 


guess the next step is: What more could be done with this data? 


At the moment, it’s in the very two-dimensional view, either in Scholarcy as one paper at a 
time, or in Obsidian via a network graph view. But what else could be done with this data? 
And maybe some of you have some suggestions about how that could be visualised, perhaps 
using virtual reality or some other means. But really, the motivation for this was to make it 
easier for me to read all the papers that I had to read for my PhD, make it into a friendlier 
format that I could, for example, read on my mobile phone. This tool called Obsidian has an 
iOS app and I can actually read this paper in a nice friendly format that will be responsive, 
including the tables, because Scholarcy also converts the tables in the PDFs to HTML. So I 
can get all that data out as well and read that on the go, which was the goal of doing this, 
really. So that’s Scholarcy in a nutshell. It exports to various other formats as well, so if I 
really wanted to export my paper, imagine this was my own paper and I want to export it to 
PowerPoint, I will turn that into a presentation. That is one thing I did do with the chapter of 
my PhD, was turn that into a presentation. I could just export this as a PowerPoint slide deck, 
and it will summarise this paper and distil it down into a series of slides. But that is the goal, 
really, is to be able to convert and switch between different formats without having to worry 
about whether did it start off as a PDF, or was it a Word document, or was it something else? 
Really, every document gets turned into this standardised format that we call our summary 
flashcard, where it’s got that same structure. I was hoping to show you the PowerPoint export 
that doesn’t seem to be playing ball today. 

So that’s basically, in a nutshell, I mean there’s a free demonstrator you can try out because 
we do have a number of free tools including the reference extraction component that Frode 
was alluding to earlier, that links all the references together, and that’s a freely available tool. 
And so is this. If you want to try this out with any document, you just upload a paper and it 
does exactly the same as what I showed you in the main document management tool, but it’s 
just one document at a time. You can load a paper and it breaks it down into this flashcard 
and then you can download that in Markdown if you want and then visualise that with all 
your other documents. So there’s a whole sequence of other tools, as well, that we have, but 
this is the main one. It’s what we call our Flashcard Generator. 

So, yeah. That’s it, really, in a nutshell. Let’s maybe bring it into a discussion and get some 
feedback, really, because it’d be good to know about how we could take the structured data 
and do something else with it other than put it into Obsidian, or Roam, or other Markdown- 


aware tools. Maybe there are some more interesting things that could be done there. So, I'll 
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pause at this point. 


Dialogue 


https://youtu. be/pdVHOoh-EL8?t=1768 


[Frode Hegland]: It really is a nutshell, and it’s just amazing what you have done. And you 
presented like you’ve done a few Lego blocks, and that’s about it. It’s just a British 
understatement. But there’s one thing before we’re going to proper dialogue I'd really like to 
see more of. And that is the bit where you come across a citation in a document and you can 
click on it to find out, to put it crudely, its value or relevance. Would you mind showing that? 
Because when you do these big graphs, where to go next is always a huge question and I 
think this, navigationally, really helps. 

[Phil Gooch]: Sure. So, let’s look at that same paper that I was looking at earlier. If we’ve 
got a citation, we can mouse over it and we can click and go to that paper. But in terms of 
what is the value of this citation, we’ve partnered with another start-up called syte.ai, that you 
may be familiar with, and what we can do is show the statistics that sites have gathered on 
every citation. And I think we’ve got a huge database now, we’ve got about a billion 
citations. What this shows me is how many other people have, not only just cited this study 
but how many people have agreed with it. So this had about 1,258 citations, but of those, 18 
have been confirming the results of this study. What that means is if I click on that link there, 
it’s going to give me some more background. Here’s the paper by Stam that this author cited 
here. And we can see that it’s got 18 supporting citations, and three contrasting. Let’s see 
what that means. It basically means that they’re saying things like, “Our results agree with 
previous studies which include Stam, and so on”. Consistent with this guy, for example. 
Basically, 18 of these citations are ones they’re all saying, “Yeah, we found something 
similar.” But three of these, they found something different. So what do they say? Well, we 
can just click on that. So, previous studies, blah, blah, Stam. Looks a bit ambiguous to me, 
not sure if it’s definitely contrasting or not. This is the thing with machine learning, 
sometimes you don’t get it quite right, and it’s a bit borderline if it’s actually contrasting. But 
you can see that, again, the context in which the other people have cited this study, that these 
guys have also cited it. Were they positive about it? Or were they negative? And so, syte.ai is 
a really cool tool for showing you this context about how everyone else has talked about this 
paper. So, for example, in this paper here, we could find out who else has spoken about it. 
Because these relationships go in two directions, we want to know what this paper is saying 


about other people’s studies, and what other people have said about those same studies. But 
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also what other people say about this study itself that I’m reading. If I click on this link here, 
it should take me to what other people say about this van Lutterveld paper, and we can see 
that actually people are a bit neutral about it, there are 31 citations and they just mention it, 
but none of them are contrasting, and none of them are supporting. They’ ve cited it, but they 
haven’t really said anything positive or negative about it. So syte.ai is a really cool tool that 
just lets you explore those citations. And we link to it as a matter of course. So every citation 
in here should have a button where you can see those stats. And then the other thing we try 
and do, and this doesn’t always work is, say, “Well, rather than going, looking, and reading 
all these cited papers, can we just get the gist of them?” We have a little button here that will 
go and find each of those papers and it will just do a quick summary of what was done in that 
paper and then we can see. It’s like a subset of the abstract, effectively. What was this paper 
about? Is it something that I’m actually interested in going and reading more about? If I am 
then I can click on it and go and read it. So the idea is to bring all that information, from each 
of those studies, into one place, either with citation statistics from the site. Again, this looks 
like a reliable study, 13 people have supported it, so that looks good. But what did it say? And 
again, we can just click the findings button here and it will go and try to pull out what the 
study found. And there are some of the findings there. So that’s another aspect of 


what Scholarcy does, that citation linking and classification. 


[Frode Hegland]: Question to everyone in the group before I do the hands up thing. how 
amazing is this? It is absolutely amazing, isn’t it? And also, the way that Phil works with 
other APIs from other services. The way these things can link together is just so incredibly 
amazing. And I don’t think most academics are aware of it. Because you’re the newest in this 


session, Ismail, please you go. And then, Peter. 


[Ismail Serageldin]: Thank you. You probably are very familiar with the work of David 
King and a few others at Oklahoma State. I was quite interested in their work a few years 
ago, because they had done hermeneutics of Islamic and Quranic work on 12,000 things. 
Phil, you had this diagram with all the nodes connected with the greens, and you said, "Well, 
where you go from here?" It all looks pretty much like one big tapestry. What struck me 
about what David King was doing at that time was that, they were able, and this was really 
stunning for me, able to put all the authors and then, surprisingly, the graph tended to group 
authors together. So, all of a sudden, the group of these Israelite debaters, back in the 10th 
century were all in one part of the graph, and all the Ash'ari were in another part of the graph. 
And the schools of thought, somehow, emerged out of that. So it didn’t look exactly flat, like 
the diagram. Based on the diagram of this thing, they were able to group them into, 
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maybe the citations, maybe other things would be able to assist in that, but if it did that, 
then you might see schools of thought emerging in the pattern in front of you. 

[Phil Gooch]: Yeah, that’s great. I think there’s a huge amount like that, that could be done. 
So that network that I showed was in another tool, in which I’m not involved in, it’s called 
Obsidian. And I’ve just put a link to it in the chat. So it’s obsidian.md. And that is just the 
tool that allows you to visualise these relationships. It’s quite basic, and it doesn’t show, I 
don’t think it can show those levels of annotations that you mentioned that David King 
showed, where he had the authors, and so on. But there are other tools that do a bit more than 
this, along the lines of what you’re suggesting. And one is called Connected Papers, which 
T’ll put in the chat, where they do try to find out similar schools of thought. The idea is, you 
put in one seed paper and it will find other related papers, not ones that are related by 
citations, but also similar themes. I will also quickly share my screen to show you. And I 
think that, what they’re trying to do at Connected Papers, is trying to generalise what you 
were suggesting, what you’re talking about with David King, where they show, here they’ve 
got the authors for a given paper and what they’ve tried to do is show related papers where, 
maybe other people have cited them together in a group, or they’ve got similar themes. And 
so you can click on each of those and find out more about them, and you’ve got the abstract 
and so on. And there’s another one, there’s quite a few tools like this, there’s one called 
Research Rabbit, which is pretty cool, but unlike Connected Papers, it only works if you’ ve 
got an academic email address, which I don’t have anymore. But those of you that have, you 
might want to check out Research Rabbit because that tries to do that. So in answer to your 
question, Ismail, there are other people doing those visualisations and trying to generalise 
them. It’s not something that I’m going to do myself. My role, really, is just to build tools that 
do convert from one format to another, so that other people can do those visualisations. But, 
yeah. I think it’s a great suggestion. And I think the potential hasn’t really been fulfilled of all 
this visualisation and linking yet. Partly because, when the data sets become large, it does get 
hard to then keep track of all these nodes, edges, and what they mean. I think, Mark, you’ve 
done some work on this with citations, showing things about who’s citing it and who cited it 
by, looking at alternative ways of doing it, other than a network graph. But I think there’s still 
room to come up with some new type of visualisation that would show all those relationships 
in a compact way. But you guys know more about the people that are doing that and me. I’m 
an NLP person. I’m not a visualisation person. So I’d love to hear more about those that kind 


of work. 


[Peter Wasilko]: I was just wondering, have you received any pushback from any of 
the Scholarcy publishing houses complaining about your personal document? 
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[Phil Gooch]: No, because I think... That’s a good question. We haven’t had any complaints 
because we’re not making those converted papers publicly available. So it’s a tool like 
Dropbox or Google Drive. You drop your papers in, you’re the only person who has access to 
those condensed versions of those papers, those interactive versions of papers. We’re not 
putting them out there in a massive database that everybody could access. So, no one’s 
complained about copyright breaches because it’s really only for personal use. But I think 
there could be a lot of value in taking every open-access paper and putting it into this kind of 
structured format and showing how they’re connected. And I think, if we were to do that, 
then, yes, publishers would complain. But we are in discussions with some publishers about, 
maybe, doing it on a subset of their papers, in some way. But it’s just a question of priorities. 
There’s only me and one other person working on this at the moment. So it’s about where do 
we spend our time, and publishers are a bit of a distraction at the moment for us. So we’ve 


had one or two conversations, but yeah, they haven’t complained, basically, is a short answer. 


[Peter Wasilko]: Ah, that's encouraging. Also, have you taken a look at the bibliometric 
literature? 

[Phil Gooch]: The bibliometric literature? I’m familiar with some of it. But not massively, 
no. I know there’s lots of stuff about the whole open citations, thinking about making every 
citation open. There’s the open citations initiative, but did you have something in mind, 
particularly? 

[Peter Wasilko]: Just that there’s like a whole subset of the information retrieval literature 
looking at co-citation relationship and term clusterings amongst society documents. And also, 
there’s a whole sub-community that’s been poking at those statistics for quite some time, and 
you might be able to find some useful connections there. 

[Phil Gooch]: Yeah, I’ve talked to a couple of people. There’s a chap called Bjorn Brems, 
and there’s also Bianca Kramer and David Shotton, who’s in the open citations initiative. I’ve 
had a conversation with some of them. We’ve actually created an API that some of them are 
actually using within open citations, I’Il just put it in here, to extract references from papers 
so that they can be connected together. Because one issue with citations, although it’s not so 
much of an issue now as it was, was that these citation networks were not freely available, 
publishers weren’t making them available unless you signed up to Web of Science or Scopus. 
But now more publishers are putting their citations into Crossref, so that people can do those 
kinds of network analysis that you mentioned. But we’ve also created this tool that other 
people can use, authors can put their own papers in there or pre-print, and they will extract 
the references and then they can be used. Some of the people at open citations have used this 


API to do some of that extraction. We’ve made that freely available for anyone to use as 


456 


much as they want until the server falls over. It’s not on a very powerful server. But, yeah. 
There’s a lot of work going on in this, but it’s not something that I’m personally involved in. I 
focus more on the data conversion side, and then, once that data is converted, I like to give it 


to other people to actually do the analysis, if that’s what they want to do with it. 


[Peter Wasilko]: Also, have you considered applying your tool to bodies and source 
codes like, throw to GitHub and look at all of the citation relationships that actually take the 
form of code inclusion? 

[Phil Gooch]: No. It’s a great idea, though. No, I haven’t done that. That could be a good 


project. 


[Brandel Zachernuk]: This is a really cool tool. I’m really excited by the idea of 
rehydrating things that are essentially inherently already in hypertext and just making 
them navigable in the way that they should be based on that conceptual content. In 
terms of suggestions or questions about further directions, the main question that I would 
have is what drives my work, so it hopefully doesn’t come across as offensive: What is the 
point of the functionality? What are the intentions that people have that they follow as a 
result of using the system? And in particular, when somebody is good at using the tool, what 
are the primitives that they establish mentally and procedurally that drive their behaviour and 
action within it? And then beyond that, what are the ways in which you can render those 
primitives concretely in order to make sure that the use of the tool intrinsically lends itself to 
understanding things in the way that an expert does? 

So, right now, you have a lot of things in it that are useful, but they’re not especially 
opinionated about what you do with them. And so, the suggestion I would have and the 
question is: How do you ramp up that opinionation? What are the ways in which you can, 
more strongly, imply the things that you do with the things and the way to read the specific 
things? 

So there are numbers, like the confirmations and address become the contrasting results 
and things like that. What do those mean and how can people understand those more directly, 
if they need to? One of the things, as these folks have heard me bang on about, I am not from 
academia, so I’m not familiar with the sort of, people’s relationship with academic papers and 
what people spend their time doing. Something that I have spent time in, within the context 
of academia is, debugging my friend’s prose. So where is (indistinct) in neuroscience, and 
I’m sure it’s not peculiar to his discipline, but you can end up with an incredibly tangled 
prose, where it’s, essentially, trying to do too many things in a single sentence, because 


there’s a lot to get through. And the sort of approach that I take is very similar to... Have you 
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ever heard of visual syntactic text formatting? It’s a system of breaking sentences and 
indenting on prepositions and conjunctions. Oh, you built it into Liquid (Author)? Right. 
Yeah, you did too. And it’s basically taking something more like code formatting and turning 
something that I think is pretty generally the case to academic text, that it can end up pretty 
hard to read. And so that it allows you to follow individual ideas, and understand the regards 


in which they’re nested and indented through that. 


So, I guess, what is the hardest stuff to do with these academic texts? And then also, 
potentially, I’m sure you’ve read and reread “As We May Think”, Vannevar Bush’s book, 
paper, column, article in 1945. Have you read it before? In large part kicked off the idea of 
computing for everybody who does computing. And it was made by the man who was 
responsible for the National Science Foundation during the American War Effort. And he was 
then complaining about the impractically large body of knowledge that was being produced 
year on year. And needing some memory extension that would allow him to navigate all of 
the, I think, academic papers, be able to create hyperlinks between them, and have some kind 
of desktop environment for doing. It’s a really wonderful read, because he’s basically 
describing the modern desktop computer, except built out of gears and microfiche, because 
that’s what his mind was thinking of in the 1940s. The reason why I bring it up and belabour 
the point is because one of the things that were really wonderful about Bush’s conception of 
it is that, the navigation of the information was just as important as the information itself. 

One of the things that I would be really curious with is, in terms of somebody’s use 
of Scholarcy to navigate the Docuverse, what are the artefacts that might be kind of re- 
rendered themselves about somebody’s consumption and processing of a series of 
documents? Because it strikes me that the browsing interacting behaviour that somebody 
engages in within the context of your system and framework that you have set up, could itself 
be a valuable artefact. Not only to the individual doing that navigation, but potentially to 
other people. Bush envisioned people being trailblazers, constructing specific trails for other 
people to navigate. Where the artefact was solely the conceptual linkages and navigation 
through those specific documents, which I think is something that Google essentially is able 
to leverage in terms of making page rank. But most other people don’t have access to it. But 
your individual trip, and the traversal of people, actually, between pages is one of the major 
indicators of what are going to be good Google search results. They have the benefit to be 
able to make use of that data, whereas other people don’t. But in your case, because you are 
particularly interested in the individual, the user making the connections, and drawing it 
between that actual browsing history, and navigation through specific things, it strikes me 
itself as a very useful artefact to see what people have missed, what people have spent their 
time on, and things like that. But, yeah. Really exciting work. 
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[Phil Gooch]: Great. There are some really great questions there. To answer them briefly, the 
first motivation and use case for this was my own need to understand the literature in 
what my PhD, which was in health informatics. So the actual idea of linking all this stuff 
together, at the time, wasn’t there. It was, actually, can I break this single paper down into 
something I can read on my iPad without having to scroll through the PDF in tiny 
print? Can I turn this PDF into interactive HTML? So it’s really much focused on, what 
can I do with individual papers to make them easier to read and digest? And what we started 
hearing back from users was that, actually, particularly novice users, novice academics, I 
should say, most of our users are people doing master’s degrees, or maybe in the first year of 
their PhD, where they may not be used to reading academic literature, and it takes them a 
couple of hours or longer to go through a paper and figure out what’s going on. People tell us 
that it helps them reduce the time by, as much as 70% in terms of understanding the key ideas 
of the paper and just being able to follow up on the citations and the sources and so on. That 
was the prime motivation, just to really make the reading experience easier. 

And, in fact, as recently just at the beginning of this year, we’ve been awarded the 
status of assistive technology by the U.K. Department of Education, because we’ve got a 
large user base. People who have dyslexia or attention deficit type disorders, where they have 
specific needs, they’re in university and find it’s hard to deal with an overwhelming amount 
of information in one go, and they really find it beneficial to have it broken down. And 
there’s a lot of research on this, in terms of, generally, why students don’t read the literature 
that they’re given by their lecturers or by their educators. They enrol on a course, they’re 
given a long reading list, and then, they have a lecture, and they go to the next lecture, and 
the lecturer says, “Okay, who’s read the material?” And most people haven’t. And 
educators have been tearing their hair out for years trying to figure out how do we encourage 
people to read. And there’s some research on this about what will encourage students to read, 
and basically, it’s: break the information down, make it more visual, make it more interactive, 
highlight some of the key points for them. Just give them a bit of hand-holding, if you like. 
And so that’s what the technology here tries to do. It provides that hand-holding process. But 
in terms of the linking of everything together, that’s a bit of a late addition, really, to 
Scholarcy. And it was really motivated by the fact that, I noticed there was a big academic 
community on the Discord channel for this tool called, Obsidian, where people were saying, 
“Well, how can I incorporate all these tools into my research workflow?” 

The big need that most researchers, or most students, anything from masters level 
onwards, the big tasks they have to do is, they have to write a literature review that 
justifies the existence of their research. What have other people said about this topic? And 
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then, when you write your thesis, or you write your essay, you’ve got to say, “Well, all these 
people said this. This is my contribution.” So the task of doing literature reviews is an 
ongoing one that everybody, every academic has to do. And so we wanted to make that 


process easier. 


Once you’ ve got those papers that you’re going to write about, drop them into 
something like Scholarcy, and it’ll break them down, and you can export them into a tabular 
format. So, one of the things I didn’t show is the export of everything to this, what we call, a 
literature review matrix in Excel, where, basically, you have about 100 papers say in 
your review, and you want to compare them side by side. That was one of the other 
motivations for building it. It was to do that side by side comparison of papers, which I can 
quickly show you, actually, while I’m talking. So, yeah. Writing literature, some people in 
academia, there’s this whole department that is just writing literature reviews. If I’ve got all 
my papers here, and there are 26 of them in this case, what do they look like side by side? So, 
in Excel, here’s the raw format, I’m just going to make that a table in Excel, and then, what I 
can do is just make this a bit bigger. And then, what I can do is slice and dice the information. 
Excel has this really cool functionality called slicers. So I can say, “Right, I want the authors 
as a Slicer, I want the keywords as a slicer, and maybe the study participants”. And so, what 
we’ve got now is able to slice and dice these papers according to their keywords. So most 
academics are quite familiar with tools like Excel. Let’s just look at all the papers that had 
112 individuals or 125 participants, for example. And we can just show those. Or look at all 
the ones that are about cerebral palsy or DNA methylation. So we can do that quick filtering 
of papers and compare them side by side. And obviously, I can make this look a bit prettier, 
but the key idea is being able to filter papers by different topics or by numbers of 
participants, for example. 

We typically want studies that have a lot of participants, and ones that only got eight 
subjects, for example, maybe aren’t going to be as useful to us. So that was the other 
motivating factor and this is how people use it to help with their literature review. So the 
whole thing about linking everything together, as I showed you in Obsidian, is a relatively 
new development if you like. And so, yeah, I’m open to hearing about how people might use 
this. At the moment I don’t think many people are using it for this kind of linking. They’re 
mostly using it for reading, and they’re mostly using it for creating these matrices that they 
then use to help figure out the literature and what’s going on. For example, you might say, 
“Well, I’m only interested in papers that have open data availability”. So I can just look at 
ones that are non-empty, for example. So if I select all the ones that are not blank, then it 
filters those papers, the only ones that have got some open data available are the ones I’m 


going to look at. Or I might want to say, “I’m only interested in papers that talk about the 
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limitations”’. It’s quite important for studies that talk about the limitations, but not every paper 
does. So again, I can filter by the presence or absence of limitations. So this kind of 
literature review is one of the ways that people are using Scholarcy. But primarily as a 
reading tool or as a document ingestion tool. So for example, the other way I can get 
information in is, if I’m reading a paper for nature, for example, I want to get it straight in, 
while I’m reading it, I can just run this little widget that we built for the browser which 
basically will read, go away, read and summarise that paper for us. And then, we can click 
save and then it’ ll save it to our library, so I’ve got that nature paper here. Again with its main 
findings, highlights, and everything. And I can do that with a news article as well. If I’m 
reading a page in The Guardian, I can click on my extension button, and again, get some of 
the highlights, key points and links to, you know, who’s Sophie Ridge, I can click on that, 
she’s a BBC journalist and newsreader, for example. So, it does all that key term extraction as 
well. And again, I can save that. So if I’m interested in news articles, then I can also use that. 
And then the other thing that people use it for is to subscribe to feeds. So you’re probably all 
familiar with RSS feeds, which seems to be making a comeback, which is great. So, if I want 
to, I can subscribe to The Guardian U.K. politics feed, and just put our asses in front of that. 
And then, if I go back to my library and say, let’s create Guardian politics, and put in that 
feed it’s going to go away and pull in those articles and turn them into that interactive 
flashcard format for me. And I can do that with a journal article, so I’m actually subscribed to 
a feed on neurology from a preprint server called ‘medRxiv’ and it’s pulling in each day, it’s 
going to pull in the latest papers. So it’s like an RSS reader as well. So people are using it for 
that. So, yeah. They’re mainly using it as an enhanced reading tool. And there’s a tool to help 
with literature reviews. 

But the whole hypertext linking and things like that is a relatively new thing that we’re 
not quite sure how many people are actually using to create those relationships between 
things. While I was talking, it’s gone away and just started to put in those Guardian articles 
here. So, I put in Guardian Politics and already it started to pull in those articles here. It 
doesn’t just work with PDFs, it works with news articles as well. So it tells us more about 
Grant Shapps, he’s the Secretary of State for transport. People use it if they’re new to a 
subject. If I’m new to neurology, I want to know what some of these terms mean. We’ve got 
these hyperlinks to Wikipedia, so if the Akaike information criterion is unfamiliar to me, I 
click on that and it tells me what it means. I’ve got the Wikipedia page about it. If I don’t 
know what basal ganglion is, I click on it and it tells me all about it in Wikipedia. So that 
level of linking is something we’ve had right from the beginning and this is well used by 
people who use Scholarcy. But this kind of graph view is not really well used at the moment. 


And we’re trying to figure out how to make this friendlier, because we have to do this in a 
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separate application at the moment. But the Wikipedia linking is very popular. So the basic 
level of doing those key concepts, and their definitions is certainly something that people use 


to get up to speed on a subject if they’re quite new to it. 


[Brandel Zachernuk]: That’s awesome. In terms of the use of things like site linking to 
people and concrete entities like the basal ganglion, I would love to see in the direct 
adornment and representation of those entities within the document that you have as being 
reinforcing the category of things that they are. So, having a consistent representation, for 
example, of people so that you have, if available, a thumbnail, but otherwise some 
indicator that these things are definitely people, show them, rather than concepts. I saw 
that you have a little bit of, it’s being able to pre-emptively pull a little bit more information 
about what you’ll find behind those things. One of the things that I really love to do is make 
sure that people minimise the surprises behind clicks so that they have the ability to anticipate 
what kind of content they’re in for. And that helps frame their experience because hypertext 
is very valuable insofar as it allows you to navigate those things. But if it’s anybody’s guess 
what’s behind them, then that can be very distracting. Because it means that it’s difficult for 
them to process things in those flows. Another thing that I’m really excited by just looking at 
that, natural language processing lends itself incredibly well, it’s a question answer and 
agentive mediated action and stuff. Have you played with the speech-to-text and the text- 
to-speech engines within browsers in order to be able to create conversational agents 
and participants? And it strikes me as a lot of fun to be able to do, where you could actually 
ask pre-formed questions of a certain kind about your corpus, in order to be able to do things 
like that. 

[Phil Gooch]: Yeah, that would be a great idea. I know there are some other tools. There’s a 
tool that does some similar stuff to what we do, it’s called Genei and they have a question 
answering thing. We haven’t done that kind of thing. But, yeah, certainly something we could 
add. Either you may type in a question like, ““What is the best evidence that supports the use 
of this particular drug against Covid-19?” for example. And then, it would go and search all 
those documents and show you which ones generally support the use of that drug, for 
example. We could do that. And that could also be a speech-type interface. So, yeah. That’s 
something that we could add to it, certainly, as a future enhancement, that’s a great idea. 
[Brandel Zachernuk]: The other benefit of a speech primary environment is that you have 
the opportunity to use the visual feedback as a secondary channel, where you can say “I’ve 
found these documents and they are here”. And then the documents are up here and things 
like that. But, yeah. It’s super cool. One of the things again that strikes me, that you’re doing 


with it, as well, is the academic paper format is very curious and very dense, in no small part, 
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because it’s for shipping important information on (indistinct). And so, as a general concept 
being able to be a little bit more generous with the space, in order to be able to characterise 
and categorise the different things that are in a paper, is a really good viewpoint perspective 
on what it is that you're able to do. Because, like I said, even though an iPad is a smaller, in 
many regards, device than the papers that you’re going to be reading, or especially a phone, 
you do have the ability to renegotiate the space, the real estate that’s devoted to those things. 
And, yeah. Being even more generous with the space that you use to carve out the, this thing 
is this, that thing is that, might be a valuable way of playing with all of the different elements 
that you’re presenting. 


[Phil Gooch]: Yeah, that’s right. That was one of the main motivations. To reduce that 
problem of squinting at PDFs on screen. Because they were meant for print. But 
everyone’s using them as an online distribution format, as well, which wasn’t what their 
intended purpose was. And so, just to try to transform that content into something that was a 
bit easier to read on screen, I don’t think we always succeed. And I think, actually, within the 
academic community, there is this, people are trying to move away from PDFs as a means of 
distributing knowledge, but people are still struggling to get away from that format for 
various reasons. Which is a subject for another discussion, perhaps. But, yes. 

[Frode Hegland]: For a long time. But since you just talked about the provocative three- 
letter word, PDF. It is something we’re discussing here. We use it archival, and we accept 


academics use it, but as an intermediary in rich format, clearly, it’s not up to snuff. 


[Mark Anderson]: Well, first of all, thanks so much. Fascinating to see Scholarcy again. It’s 
something I’ve been meaning to find some time to dive into again. Because it's interesting 
you talking about the Obsidian graph and things. So, for instance, one of the problems there 
is what it actually does. It shows you the links that you made. When I say you made, now this 
gets to the interesting part. If we begin to do automatic extraction, who made what link for 
what purpose, this is where we get lost. And there’s a massive, I mean, obviously, there’s 
Obsidian and Roam and there’s a whole cult around zette/kdsten. But a lot of these things, 
unintentionally, is the ‘underpants business gnomes—the Underpants Gnomes business 
theory, where if you collect enough stuff, a magical thing will happen, and success at the 
end, and no one quite knows what the magic is. I think one of the interesting challenges, but 
opportunities, actually, to the data set you’re now sitting across is to be able to start to surface 
some of the relationships. The real privilege you have with the dataset is that you know 
what’s there, and you can begin to make more objective study comments as to what the links 
mean, that many people can’t. So some interesting, in a sense, research to be done there. So 


one way one could look and try and make sense about diagramming would be, to take an area 
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that we know has been just really well trampled by people, so you might say, “Well, there are 


a few surprises in the literature”. And then play around with the visualisation. 


Because you want to be able, to then, have something that’s otherwise really hard to do, 
saying, “Okay, I’ve made this wonderful-looking thing. Is it meaningful or not?” And most of 
the time we do this, we just don’t know. The main thing is we know it looks pretty. And that’s 
another problem because we like to make pretty and aesthetically pleasing graphs, whereas 
life would tend to suggest that the messier it is, probably the closer you are to the ground in 
truth. So I think that might be an interesting area to look at. I think, then, to make sense of 
what the either inferred or extracted hypertextual nature or linkage in the data is. It’s probably 
most meaningful to take, or most useful to take a bit that’s essentially well known for 
whatever reason. But one where there isn’t a great thing. So, don’t pick something that’s a 
great topic of anxiety, or social warfare at the moment. But I think that there ought to be 
places where we can see this. Which brings me on to another thought which is the degree to 
which I’m guessing that the sciences, the paper in the sciences are more tractable to this 
process than the arts. Because the language is, by and large, more direct. So we’ll talk about a 
thing, and that’s the thing that we can go and look up, whereas, in a more pure humanities 
side, the reference is just maybe elliptical, and did you have to know the subject matter quite 
well, to know that they’re actually referring at one form removed from the subject that’s 
actually under discussion. I think that’s just a state of where we are with the art, rather than a 
limitation, per se. But is it the case you get more back from science areas? 

[Phil Gooch]: Yeah, that’s right. It does. For the reasons that you mentioned, the structure 
tends to be quite standardised. They have what they call the IMRAD format, Introduction 
Methods Results And Discussion. They’re very much about stuff that can be packaged neatly 
into facts if you like, or the factoids, or things that got some evidence about it. Well, we have 
tried using it on certain subjects in the social sciences, things like philosophy and biography, 
and well, I mean, literature generally, as you go towards the literature, and particularly 
fiction, it doesn’t really work at all other than the fact that we can pull out named entities 
from people and places and so on. But in terms of pulling out the argumentation structures, is 
much harder in the humanities. But interestingly though, actually some of the feedback we’ve 
got from some of the users is that, in a more social sciences subject, it does really well. And 
less well in things like philosophical and rhetorical-type articles, in the hard sciences, or the 
stem sciences. It doesn’t do very well in engineering. And I think the reason for that is a lot of 
mathematics, and we don’t really handle mathematics very well. Getting decent mathematics 
out of PDF is hard. And then often, an engineering or mathematics paper is all about the 
equations, and the discussion around it is maybe not peripheral but it’s secondary to the main 


maths and formula that you’re presenting and putting forward. So, yeah. There are some 
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subjects that are harder to apply this kind of NLP to, and certainly, humanities is one of them, 
anyway. 

[Mark Anderson]: I hope you’re mentioning it because I don’t share that as a negative at all, 
I was just interested to see how the coverage goes. Because another thing that occurred to me, 
in terms of, again, because you’ve got this fabulous rich data set, one of the things I always 
find myself worried about when I was doing the research was avoiding the Stalinist theory of 
art—because 81 people said this was really good, it must be good. And indeed one of the 
things in our PhD group in Southampton was discussing was actually a way that you could 
start, for instance, to classify what’s a drive-by citation. “Oh, I have to cite that because 
otherwise I get shouted at’. And that was, to my mind, a meaningless citation, because 
actually it’s been done for no real good intent, as opposed to the thing you actually genuinely 
wanted to cite because it actually added interest. And that strikes me as a challenge when 
doing this extraction, not because of the sin of commission, but you get to the next level. So, 
in a sense, do we need to start learning new ways to read this? So as a student or a user of this 
rich data set, what are the new questions I need to learn to ask? Because, to a certain extent, 
we arrive at this technology at the moment. Sort of, “Oh, look. This number is bigger than 
that number”. And we don’t often stop ourselves from thinking, yes, but is that a deep enough 
thing? There are some interesting angles to be played there as well. How one might tease 
apart some of the raw numbers which otherwise float up the surface. Because this is what I 
was thinking with bibliographies and these raw citation counts. Because maybe it’s just a 
field I was working but I don’t think so, that I was often surprised at how many times I went 
to a really highly cited paper, I’m thinking, I just don’ see what is so special about this. And 
even when I put in the context of what was known at the time, it’s still not special. It’s clearly 
being cited because it’s getting cited a lot. But no one has ever thought to say, “This actually 
isn’t a very interesting or useful paper”. And at a slight tangent, I’m interested to know what 
do you see as an edge, as to how far back you can easily go with things? Because, 
presumably, with PDFs, you don’t get back very far before the OCR and stuff was not that 
hot. Or are you re-OCRing stuff, or? 

[Phil Gooch]: No, we don’t have an OCR engine at the moment. The PDFs do need to have 
extractable text. We did a project a few years ago with the British Medical Journal where we 
were just pulling out the end of article references from a collection of PDFs which were only 
scanned images. They did the OCR themselves, they sent them off to a company to do the 
OCR. We got the OCR versions of the PDFs back. And then we did all this extraction for 
them. And the data was really noisy, but at that time, we were just interested in getting the 
bibliography from each other. So the trouble is, we’re doing OCR on-demand, we often get 
people uploading 200, 300 page PDFs, and the idea of doing that on-demand just fills me 


465 


with fear, having that run at scale. So we don’t do that. But, yeah. It could be done but that 
would be a separate standalone project, I think, that would be a research project to go and try 


to text mine that archive if you like old PDFs. And do something interesting. 


[Mark Anderson]: The reason that it sticks in mind is it so there’s an almost implied 
temporal cliff somewhere, some distance back from us, where things start to come into easy 
digital focus. Which is unavoidable, but it’s perhaps something we need to start to recognise. 


Yeah so there was one other thought but it’s passed from mind, so I’Il let that be. 


[Frode Hegland]: So, Phil. The reason you are here, as we’ve discussed before, is you allow 
for analysis, for interactivity. And I’m wondering, before, actually, I’m going to ask the 
question first, not to you, actually. Brandel and Fabien. I’m going to waffle on it for a minute 
now, but if you guys have something you want to show Phil that you have worked on or 
something else in VR, to help him see where this fits. I just want to highlight, for my own 
personal work, with my own personal software, when I look out and I see so many people 
doing amazing stuff, the only thing that I’m trying to contribute to is simplification. Because 
you can make things really horrendously complex, obviously. So I’m wondering if, maybe, 
by making interactions with this more tangible, we can have more... Yes, here we go. I can 


stop waffling now, Fabien would like to show something. 


[Fabien Benetou]: Hey, everyone. So this is not actually a network analysis, graph analysis, 
or any Scientometrics. Simply putting the PDFs in space of an upcoming conference, it was 
for a VR conference. And then I think a lot of people got that struggle, a lot of people look 
interested, but then you have to start with one. I know, at least I can’t read two or ten papers 
at once, so I need to find which one. And basically what I do is, I put them in space, I set up 
the space to make it friendly or wanting with the conference. And then I’m going to put them, 
I have a little annotation system with a 3D object where I put a post-it note if I need to write 
something on it if I’m not sure if it’s interesting if it’s really mind-blowing and I want to read 
it first. And, yeah. That’s the result. It’s a social space, so I can invite somebody to go through 
and then we can discuss which one to read first. And then, at the bottom right, I don’t know if 
you can see clearly, there is a grey platform, and then I can send it to my ink reader and 
writer so that I can sketch on top and update it and all that. I have a couple of other examples 
where it’s more the graph view. And then you can go through it, but it’s a bit more abstract. 
So I think this was the more tangible way, and I would definitely like to have my personal 
annotation through this, for example. But I could very easily list to next to a paper or an 
article, information related to it. For example, scaling based on popularity or anything like 


this. Just a simple example. 
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[Phil Gooch]: That’s great. Yeah, that looks like a really nice way of navigating and picking 
out which sections you want to read, and papers you want to read. What I was looking at 
when you were showing that, just reminded me of a paper I saw years ago called document 
cards, which was one of the motivations for Scholarcy. Where they turn each paper into what 
they call ‘Top Trumps’. So, if you’ve got a lot of papers to visualise, it turns each paper into a 
single graphic that’s got the main image from the paper and maybe a couple of quotes from 
the paper. And it’s a way of showing everything on a paper in a single thumbnail. And maybe 
there’s a way of doing something like that, instead of showing those PDFs in your virtual 
reality, you’re showing, maybe, a condensed version of them, that maybe has just enough 


information to decide whether you want to read it or not, perhaps. 


[Frode Hegland]: That's definitely worth us having a good look at just a little bit. Phil, on a 
sales pitch for the whole VR thing: How long ago has it been since you put on a 

headset? More than a year? Because, Phil, you must have done some VR at some point, 
right?. 

[Phil Gooch]: I’ve not done anything. I might have put on a headset in a museum once or 
something, but... 

[Frode Hegland]: Because the key thing is, it’s nothing like Second Life at all. And what 
Fabien was showing there is, once you're in that space, it becomes really useful and 
navigable. I sometimes write using Author, my own Author in VR. For the opposite reason 
that it’s normally good for because it means I have a limited field of view, I have a nice 
background, I have a decent size screen, the visual quality is good enough for writing, and it’s 
good enough for reading. You wouldn’t want to read forever. Sure, absolutely. But where the 
whole system is now is that we’ve done some experiments of a mural, and just having a 
single mural is absolutely amazing. Because it is really hard to describe, when that mural as 
an image, is on a computer screen, you kind of move it about, yes, of course, you can do that. 
But when you can have it huge and then you do a pinch gesture and it comes towards you and 
you zoom in different things, it’s kind of not explainable why it’s so special. And one of the 
reasons Ismail is here, we’re looking at doing some mural and timeline related to Egyptian 
history. It is really hard for us, we only started. I mean, Fabien and Brandel have been going 
for a long time, but the rest of us, we only started, basically in January. So I have my headset 
here, goes on and off depending on what we’re doing, but it’s really hard to explain the point 
of it. Because sitting down VR is one thing, but what really brought me over the edge was 
when Brandel said just moving your head a little bit as you naturally do, it changes 
everything. When we have meetings in VR, which we sometimes do, the sense of presence 


and being with other people, because the audio is spatialised, so if someone’s sitting there, 
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the sound comes from there, it’s absolutely phenomenal. 


So, I really think that Obsidian and all of that it’s nice, and even, as you saw in the 
beginning, Mark has taken, not even that many documents, but enough documents that that’s 
all the system can do, into this space, it quickly becomes messy. So, I think what you 
contribute is the ability to change the view rapidly and intelligently. There are so many 
interfaces for VR, and a lot of them is about using your hands, grabbing, and moving, and 
that’s all well and fine. But in some of them, you can have literally buttons to press for 
certain things to happen. So, I could easily imagine a document space, you start with one 
document, and at least in the beginning, you have a huge set of buttons underneath, very 
inelegant, obviously, that when you come across a citation, you can do what you already 
showed. They can start growing the trees. But all these buttons, again, initially can help you 
constrain and expand that view. It would be nice to have a spoken interface, it would be nice 
to pull, and that needs to be experimented with. But the reason I was so excited to have you 
here today was and is the real interactivity that you give. You take data that’s out there and 
you make it tangible in a whole new way. 

So I hope that, what we’re trying to do, we’re trying to do some sort of a demo for the 
next Future of Text. We’re looking into building some work room. And Brandel has already 
taken, from Author, because Author documents are dots. They’re called dot Liquid, like, dot, 
dot, dot, Liquid. Inside them, we have JSON. So we have some of those goodies already. 
He’s been able to take the map view, with the relationships into VR. And, of course, it’s 
relatively static, but you can already touch things and see lines appear. To be able to go 
further, and to do, with what you have made available, would be really quite exciting. I mean, 
I could very well imagine doing the reading you’re talking about. You talked about making it 
fit on an iPad, but what about making it fit a whole room, right? Just putting one wall, to 
begin with. Where do you actually put the pictures? Where would you put the graphs? So 
many questions come up. It gets really interesting. It’s not very obvious at all. But thank you 


for allowing us to think with available data. 


[Phil Gooch]: Thanks. Well, it was great to have the opportunity to chat with you all. Thanks 
for inviting me. I just wanted to touch on one thing that we spoke about in an email. Because 
at the time I had a hard time thinking what is the VR/AR angle on this. But you can imagine, 
in an augmented reality setting that you might have a book or a document in front of you, and 
you’ve got this augmented view that says, “Actually, here are the main people citing this 
paper. This is what they say about it. Here are the main findings of this paper” as a separate 
layer. So you’ve got the paper there you can read and navigate in this 3D space, and you’ ve 
also got this layer that says “Hey, here’s the really important stuff in this paper that you need 
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to know. And this is what other people are saying about it”. And maybe that’s one of the use 
cases for AR in this kind of idea. And as you know, Frode, the API is open, so if there’s 
anything that you want to add for your demo, just give me a shout and we can make it 


available to you. 


[Frode Hegland]: I'll go over to Brandel, but just really briefly, the thing about how things 
are connected in a VR space is really up for grabs at the moment. It really is the wild west. So 
one thing I think we need to do now is, just dream crazy dreams. For instance, the Egyptian 
opportunity, let’s say you have the mask of Tutankhamun sitting literally on your desk as 
you’re working on a project, you should then be able to say, “Show me how that relates to 
timeline, when it was found, and when it was used. Show me that geographically”. All these 
things should be able to come together. And right now, other than some idiot on Zoom, me, 
doing it with his hands, it doesn’t really connect. And I’m hoping that your, first of all, your 
parsing and your genius, but also your willingness to open your APIs to others, and to use 


other APIs can be a really powerful knowledge growing hub. And, yeah. Brandel, please? 


[Brandel Zachernuk]: Thank you. Yeah, I definitely echo everything Frode is saying. If I 
can characterise what it is virtual reality, augmented reality, spatial computing at large does is 
that, when you have a display, be it a phone or a screen, even if it’s 30 inches or whatever that 
is, it’s still very much performs the function of a foveal vision. The central vision of what 
you're looking at. And there was a lot of really neat exploration of the practical cognitive 
consequences that in the 1980s, where they’re saying, “It’s like browsing a newspaper 
through a hole the size of one column wide”. And what virtual reality does is take that filter 
away, so that you’re able to read those newspapers but you’re also able to see the whole 
space around it. And to that end, I think we, unfortunately, as a result of having 50-odd years 
of computing being the primary mode of interaction for at least some information knowledge 
workers, and certainly 30 years of it being in the absolute dominant form, is that we have 
surrendered the space that we would typically do information and knowledge work in, to a 
small computer with a very even smaller visual real estate. So we don’t have the ability to 
think about what the entire space is for, can be encoded for. And to that end, I feel like we 
have to go back to the metaphors that spring from understanding something like a kitchen or 
a workshop, where you have tools, they have places, when you’ re standing in those places, 
when you’ re gripping things in certain ways, that means you’re doing certain things. And that 
you might move a workpiece from one place to another in order to be able to undertake some 
kind of manipulation over it. And so, my hope is that, when people can return to that, within 


the context of knowledge work, where you can say, “I’m looking at this thing right now. But I 
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have this stuff around me”. One of the things that I showed Frode and other folks in this 
group was being able to have writing that you’re doing here, and then having the word count 
over here. So you don’t have to click a button, open a menu in order to see that that 
information is available. Simply from something as simply reflexive as turning your head. 
Likewise with visual image search happening at the same time. But the other thing that this 
increased capacity for context does is that it increases, by orders of magnitude, the way in 
which scale can be used. If you think about a museum, in contrast to a book, or in contrast to 
an academic paper, which is even more compressive constrained, the way that type scale can 
be vastly changed in order to tell you things. Like, the exit signs and the titles over things are 
not just two times larger, but they’re maybe a hundred times larger. When you have a big 
piece of writing on a wall talking about how great (indistinct) is, they’re four different things 
but the sort of experiential consequences are absolutely legit. Because of the fact that you can 
devote that space to that, and this space to this. And so, yeah. I’m really excited about seeing 
all of the semantic information and insight that you have in Scholarcy, and really excited 
thinking about how to encode that into an entire space that people can manipulate and 


intervene on, at that scale. 


[Phil Gooch]: Yeah, that would be awesome to be able to do that. Our API is open, so if 
people want to try doing that, integrating it into other systems they can do that. So, thank you. 
And also thanks for your suggestions about (indistinct) those entities. What type of thing are 
they? Are they a person? A place? And so on. Like you said, clicking on those links so you 
know where it’s going to take you in advance without having to wonder. Some great 
suggestions here, so thanks very much, Brandel. That’s great. I’m afraid I have to go. Lovely 


to meet you all. And, yeah. I'll chat to you again soon. Hopefully at the next Future of Text. 
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Peter J. Wasilko 


Benediktine Cyberspace Revisited 


When we speak of Benediktine Cyberspace, we mean a 3-D visualization inspired by 
Michael Benedikt’s seminal text, “Cyberspace: First Steps” (Benedikt, 1991) and in particular 
Michael Benedikt’s chapter therein on “Cyberspace: Some Proposals” (Benedikt, 1991) and 
Alan Wexelblat’s chapter therein on “Giving Meaning to Place: Semantic Spaces” 
(Wexelblat, 1991). The main takeaway here is that a VR Environment need not simply mirror 
the three dimensions of our real world, presenting a First Person Shooter like representation 
of real or stylized spaces (which we can call Architectural Spaces), but can instead directly 


render datasets containing more than three dimensions, or attributes if you prefer. 


This can be achieved in an intelligible fashion by presenting a series of “slices” of our 
higher dimensional objects, in which arbitary object attributes are mapped, three at a time, to 
our familair X, Y, and Z axes. Since multiple objects might share these three attributes, an 
occupied point in our initial space can be thought of as holding the entire Result Set of 
querying our database for all entries that share those three values of those three attributes. 

Attributes that we choose to represent positionally in terms of the axes in a 
visualization can be said to be Extrinsic. Whereas, any additional attributes whose values we 
indicate with say shape or color or brightness or opacity of an occupied point are said to be 


Intrinsic. 


Each axis can be said to represent an extrinsic dimension that can correspond to an 
attribute or property of the objects in our dataset. How the values of attributes are mapped to 
points along an axis allows us to classify the kind of dimension that attribute represents. 

Wexelbart posits that there are two kinds of dimensions Absolue and Relative 
(Wexelblat, 1991). 

An object’s position along an Absolute Dimension is directly controlled by the scaled 
mapping of the values of one or more of its properites to that dimension. 

The location of objects along a Relative Dimension are determined by making pairwise 
comparisions of all entries using an ordering relation like greater than or after without 
necessarily knowing exact values for the attribute in question. Since multiple objects may 
satisfy a given ordering constraint it is possible for them to overlap, making any visualization 
of edges connecting such nodes unintelligible, unless one or more extra orthogonal (i.e. set at 


90 degrees to the other axes) spacer dimensions are introduced so we can spread overlapping 
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points out to view them and their connections individually. 


Since a Relative Dimension corresponds to a relation between elements we can 
subclassify it based on the properties of that relation. 

Here we are concerned with the mathematical property of transitivity, which is to say 
whether the relation’s holding between an element and a second element, and the same 
relation’s holding between the second element and a third element, implies that that it also 
holds between the first element and the third element. If this is the case, we can conclude that 
the relation describes an acyclic graph so there will be no cycles among elements and the that 
the relation can be represented in Euclidean Space. If a relation is not transitive, it may 
describe a general graph containing cycles — as in the case of the winning relation in Rock, 
Paper, Scisors which loops around on itself with Rock beating Scisors, Scisors beating Paper, 
and Paper breating Rock. Such relations can’t be represented in Euclidean Space since 
moving far enough in one direction causes one to loop around to one’s starting point. This 
can of course be represented in one dimension by cutting the loops and stretching it out in a 
line and then “warping” from one edge back to the other — as in early Video Games where 
exiting the screen on the right side would cause one to re-enter it from the left or by 
duplicating a point at opposite ends of the display range or only drawing its right half up 
against the left edge of the screen and its left half up against the right edge of the screen. 

Alternatively a graph relation can be represented From the Outside by embedding it in a 
2-D or 3-D Space. General Graphs can be represented in 2-D by drawing their verticies as 
points at arbitrary locations and connecting them with potentially crossing lines called edges 
or in same manner in 3-D without any overlapping edges. 

We can visualize a Non-Euclean Geometry From the Inside where all Three dimensions 
in a volume might represent non-transitive relations, in which case the six faces exiting a unit 
cube are logically glued together in one of a number of possible configurations mapping pairs 
of faces under possible rotation called manifolds. 

But rather than viewing the relations captured in such unnatural spaces From the Inside, 
it is much easier to visualize them From the Outside as traditional graphs in a 3-D volume. 

If the comparison relation underlying a relative dimension is dervied from values, such 
that we can determine relative distances between pairs of points in the space, we can 
compress the dimension based on these values and space points to preserve the degree of 
differnce between pairs — spacing points relative to the greatest distance between points in 
the set of all pairs relative to the minimal distance between the points in such pairs; otherwise 


we can simply give them a uniform spacing. 


Alternatively, in dealing with absolute dimensions we can place points at their natural 


locations along the dimensions and then compress or fold the space to collapse large empty 
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regions while adjusing axis labels to reflect this non-uniform metric. We can call all of these 
sorts of presentations Elastic Spaces as they will be expanded or contracted based on the 


density of their contents to make optimal usage of available screen realestate. 


We can also classify dimenions based on the type of values they can represent and how 


many points they can contain. 


Wexelblat’s Taxonomy of Dimensions 


Linnear Dimensions 


A linnear dimension will corespond to the set of Real Numbers expanding in both directions 
from an origin to hold an uncountably infinite number of points. It can also be thought of in 
terms of the output from a grammar containing repeatable productions that can generate an 
arbitrary number of strings whose lexical order will place them between any two other strings 
generated by that grammer. In other words, any grammar that can produce infinitiely 
subdivisible or refinable lists! The Grammar describing the representation of Real Numbers 
(where we can generate 1.5 which sorts between | and 2, and 1.25 which sorts between | and 


1.5, and so on up to infinitity) falls in this class. 


Ray Dimensions 


A ray dimension is also uncountably infinite, but will be bounded by an origin at some point 
on the number line and extend in only one direction to positive or negative infinity. The sets 
of Positive and Negative real numbers fall in this class as well as Age and Weight properties 
along with the output of a grammar describing Ted Nelson’s Tumblers (which can be refined 


with new dot delimited sub-sequences). 


Quantum Dimensions 


A Quantum Dimension is most similar to a linear dimension but holds only values that can be 
mapped to the Countably Infinite set of Integers, making the space granular with no 
subdivisions of “cells” being possible. Whole Numbers, Natural Numbers, Prime Numbers, 
and arbitrarily long strings drawn from a fixed alphabet of sysmbols that are sorted by length 


have this property, as do any sets drawn from a fixed pool of possible elements. 


Nominal Dimensions 


A Nominal Dimension is a Quantum Dimension that has been constrained by Domain 
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Knowledge such as the Names of Employees as opposed to a grammatical notion of Possible 
Names in the abstract. This corresonds to the invokation of a Semantic Predicate like 
“Previously Defined” in a Parsing Expression Grammar which might consult a look-up table 


to reject syntactically valid inputs that haven’t yet been declared to be recognizable. 


Ordinal Dimensions 


An Ordinal Dimension may have up to a Countably Infinite number of members and can be 
thought of as being an ordered set or more generally a list that might contain multiple copies 
of any given element (as opposed to having an element composed of multiple copies of a 
given symbol — e.g. the list of elements [ ‘a’, ‘a’, ‘b’, ‘a’ ] vs. the element ‘aaba’) in a 
fixed sequence, such that we can query the first, second, third, or fourth element; or 
determine that element ‘a’ recurs three times in the a list as its first, second, AND third 


elements. 


Functional Dimensions 


Here Wexelblat would place all attributes defined by complex forumulae whose values are 
subject to change, presuming their evaluation “at run time” as computer programmers would 
say. This aspect of the taxonomy seems a bit at odds with the others since the values 
generated at visualization time would be ammenable to classification under one of the other 
categories in his Taxonomy. So a Functional dimension is perhaps better thought of as an 
Aspect or Modifer of one of a core dimension type; or as being analog to the function of the 
Volatile keyword in the C Programming Language. 

The other weakness of this categorization is that it tends to conflate the Name of the 
Method, its Type Signature (i.e. what kinds of data objects it expects as inputs and what kind 
of value it produces) which might vary across data object (in which case the dimension would 
be most properly understood as representing / holding a Multimethod in Programming 
Language Design parlance), its Implementation(s), and the results of its Application to the 
dataset being visualized. 

It is unlikely that average system users would be concerned with the internals of the 
functions represented, so in a practical system having gobaly unique function names bound to 
code objects (holding their actual implementation code as an intrinsic attribute) would 
probably make the most sense. Alternately, we might be concerned with whether data objects 
support the invocation of a given method (e.g. Which data objects have a notion of “local 
time’’?); how that value is computed (e.g. By querying a nearby time-server and returning its 


result vs. looking up Grenwhich Mean Time from its office time-server and then applying a 
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local time-zone offset adjustment); or its current value (i.e. running the code to get its final 


value). 
Visualizing, Editing, and Navigating Benediktine Cyberspaces 
Visualization 


A system for working with a Benediktine Cyberspace will be a hardware/software 
amalgam called a Cyberdeck. On activating one’s Cyerdeck, one will be presented with a 
menu of pre-defined visualizations like “Peter’s Personal Library” as well as the option to 
create a new visualization, which would walk one through a set of dialogs to select a 
dataset or datasets of interest, which could be inspected to select one or more dimensions 
of interest and to describe how to map them to 3 extrinsic dimensions plus optional intrinsc 
dimensions. The final result will be a fully specified volumetric visualization called a 
Chamber or Space. 

At this point we can assume that even smooth linnear dimensions will be quanticized 
for display purposes, so each mathematical point in 3-space represented in the visualization 
will be mapped to one or more logical pixels on the display which taken together will 
constitute a Cell in the cyberspace containing one or more display space Voxels (ie. 
Volumetric Pixels). 

Each cell can be colored to represent up to three intrinsic dimensions or we can scale 
the visualization to increase each logical cell’s physical voxel count enough for it hold an 
arbitrarily large nested sub-visualization that might take the form of a simple stylized 3-D 
shape, a compact block of line-wrapped text, a 2-D image, an arbitrarily detailed 3-D 
model, or a nested visualization in which the walls of the cell might even be treated as 
independent 2-D display surfaces (in which case a spin affordance would let one rotate the 


cell around to see its hidden exterior faces). 
Editing 


Where a cell contains only a single data object, a grab affordance can be provided through 
which the user can take hold of the object within the cell and drag it relative to the 
visualization’s extrinsic dimensions to update the attribute values of the selected object, 
(e.g. dragging a timeline item update its start date attribute). When several items are 


present within a cell, a modifier affordance should a allow a user to select All or SOME of 
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the data objects within the cell for a grab and drag to update operation. 


Regardless of whether a cell contains any data objects, it can be regarded as holding a 
Cursor or Probe into the data-space, such that a new operation could be invoked with the 
cell selected to create a new data object with that object’s attributes to which the 
visualization’s extrinsic dimensions are mapped automatically set to those values of the 


currently selected cell. 


Navigation 


As alluded to above, Chambers can be nested to hold linked visualizations, or they be can 
entered zooming in or crossfading to visually replace the current top level visualization 
with the one contained in the selected point, or they can be unfolded to create a 
dynamically linked top-level sibling or nested child visualization of a chamber holding the 
cell’s content bound to a different set of extrisic and intrinsic attributes. When a point is 
unfolded to open a linked visualization within the volume of its parent chamber, we call 
the resulting nested chamber/cell a Subspace. 

There is a certain level of terminological ambiguity in the use of the terms Cell, 
Chamber, and Subspace with Chamber generally being use in the case of top level 
visualizations whilest cells can refer to subvolumes or subvisulizations. 

Unfolding can be applied to one or more data objects to look inside of them, or to a 
Probe (conventionally represetned as 3-D cross-hairs) to look at the result set returned by 
treating it as a Query By Example. Dragging a Probe that has been unfolded around in its 
orgin space, will have the effect of scrubbing (in video editing parlance) through the result 
sets returned by dyaamically updating the Probe’s attribute query values (based on its 
extrinic location in its origin space) to dynamically update the contents of any nested or 
unfoled linked visualizations! 

In VR, these linked visualizations could be seperately positional by dragging them 
around in the virtual environment, with perhaps glowing 3-D Bezier Curves sweeping out 


behind them to maintain a visual connection to their cell of origin. 


Comparing Objects 


In his chapter, Benedikt offers a brief taxonomy of how pairs of objects can be compared 
both within and across chambers/cells (Michael, 1991). If two objects share the same values 


for both their extrinsic and intrinsic dimensions in one or more (assuming we are dealing with 
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mutliple copies of the same top-level chamber with independent probes and sub-spaces) 


chambers/cells they are said to be Self-Same (i.e. the same underlying entity in the dataset). 


If they share the same intrinsic attributes and the same values for them, but occupy 
different extrinsic coordinates within the same extrinsic dimensions, they are said to be 
identical. If they occupy different coordinates within the same extrinsic dimensions and share 
the same intrinsic dimensions but with different values, they are similar; but if they don’t 
share the same intrinsic dimensions we say that they are different. This roughly corresponds 
to notions of class membership in Object Oriented Programming. 

If objects in spaces with different extrinsic dimensions are compared and found to share 
both the same set of intrinsic dimensions and the same values for each of them, they are said 
to be super-identical. If they share the same intinsic dimensions but with different values for 
them, we say they are super-similar. However if they don’t share the same intrinsic 
dimensions we say they are wholly different. 

These relationships are a function of both the objects in our dataset and our choice of 
how we map their attributes to intrinsic and extrinics dimensions. So the same two objects 
might be similar in one representation and super-similar in another. 

If an object is dragable within a dataspace such that its extrinsic coordinates are 
othogonal to its instrinsic dimensions and their values, the object is said to have self-identity. 
If its intrinsic dimensions are preserved with movement, but their values are computationally 
bound to its extrinsic coordinates it is said to have self-similarity. If movement determines its 
set of intrinsic dimensions, it is said to have a strange identity. 

As crazy as this might sound, it can be practically applied in a user interface where 


placing an object in a designated region changes its class / prototype as can be accomplished 


via adornment actions’° and smart adornments’’ in Tinderbox Map Views’® (Note that on a 
deep level all notes within a Tinderbox document are similar in that they share a single global 


set of potentially instantiated attributes/intrinsic dimensions). 


The DataProbe HUD — An Additional Possiblity in VR 


It addition to the aformentioned visualizations, we can also imagine providing a VR user with 
a DataProbe HUD that would have a set of 2-D or 3-D virtual display panel slots that would 
remain at fixed positions (relative to a user’s head or external environment — depending on 
user preference) to display visualizations of slices of attributes of the cyberspace cell being 


looked at, as determined by eye tracking. 


For example, one might have an employee visualization depicting the faces of everyone 
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in the shipping department and use HUD slots to show the full name, age, rating, and 
accumulated vacation days of whichever face one was looking at. Or one might be looking at 
a textual list of deparments and have a HUD Slot hold a 3-D overview of all of the 
departments by employee count, budget, and revenue with the cell correspoding to the name 
one was looking at light up in the HUD to give a perhiperal sense of how it relates to other 
departments. 

An inward swiping gesture could swap the main visualization for one in the HUD or 


vice versa with an outward swipe! 


Future Work 


Considerable work remains to be done in cleaning up the nomenclature associated with 
Benediktine Cyberspaces. It would probably prove useful to ground them in Category Theory 
and also to look at their relationship to Type Systems in the realm of programming langauge 
research. 

In this brief overview we have tried to tease out a large number of useful distinctions 
which suggest User Interface Design opportunities in VR, but the terminology will likely 
prove somewhat offputting to readers without a strong maths background, so some sort of 
Illustrated Guide for casual system users might be desirable particularly when we reach the 


point of deploying functional demonstration systems. 
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Peter J. Wasilko 


Putting It All Together 
Interface Affordances for the Serious Use of VR 


This talk revisits the ontology of Benediktine Cyberspace”? and speculates on how it can be 
extended with affordances from other areas of CHI research to produce a usable platform for 
Serious VR. 

An optimal system will support mixed initiative mutli-modal interaction between 
Spatialized Content in a VR Pane, a history of State Transitions and User & Software Agent 
Messaging in a Transcript Pane, and a Textual Dialog leveraging references to selections in 
the other panes to drive the overall system via a Command Line Interface Pane. 

In discussing the VR Pane we will first consider the nature of Dataspaces and the Kinds 
and Types of Semantic Dimensions that can be used to define them. We will also consider 
how points can represent a Query or Datum and how we can link and transition between 
visualization via Embedding, Unfolding, and Semantic Zooming. 

We will then argue for adapting the MIT Media Labs’ Chat Circles UI as the 
centerpiece of a Transcript Pane and conclude by considering how the Inform 7 UI can 


inform the design of our Command Line Interface Pane. 
Future VR Systems Should Embody The Elements of Programming 


¢ Primitive Expressions ( Data Literals & Special Forms ) that can be Evaluated to Yield a 


Value or Perform a Computational Effect. 
¢ A Means of Composition to build Data Structures & Functions. 


¢ A Means of Abstraction so those aggregates can be Named and Manipulated as First 


Class Values. 
( See The Structure and Interpretation of Computer Programs §1.1 ) 


They Should Also Provide User Interface Affordances that make Simple Tasks Easy and 
Arbitrarily Complex Tasks Possible 


Requisite Affordances for Productive Work in VR 
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A Way to Browse, Navigate, and Reference: Spatialized Content — A VR Pane 

A Way to Browse, Navigate, Reference, and Manage Attention among: Active Views, The 
Command Line History, and Messaging in a Mixed Initiative Dialog with Other System 
Users and Local or Remote Software Agents — A Transcript Pane 

A Way to painlessly and efficiently drive the system that supports references to selections 
in the other Panes — A Command Line Interface Pane 

The three panes should be independently resizable, duplicable, nestable, and positionable 
in the overall VR World and their state at any point in time should be a first class value in 


the Transcript Pane that can be bookmarked and manipulated. 


The VR Pane 


The VR Pane is the primary region for summoning up Spaces each of which can be 
imagined as being a dedicated conventional monitor, that might hold a traditional 
document, a Domain Specific Visualization, a Hyper-Othogonal ZigZag Structure, a 
Second Life or Sinespace style 3-D Chat Region, or an abstract Benediktine Cyberspace. 
We can embed an arbitrary Graph structure in a 3-D Space as a set of vertices connected 
by non-crossing edges; but if the relation represented by the graph is non-transitive so the 
graph contains cycles ( like winning at “Rock, Paper, Scissors” ), one would need to 
simulate a non-euclidean space to view it from “the inside” (1.e. to assign the relation to 
the X, Y, or Z axis for use as an extrinsic dimension). 

Cyberspaces can be Overlaid / Superimposed on one another as in AR or a Geographic 
Information System supporting multiple layers 

Spatial Layout Managers can let us call up standard tableaux of relatively positioned 


spaces that we can reference in a Command Line Viewspec 


The Transcript Pane 


The Transcript Pane lets us Manage Attention, Navigate Temporally. and Abstract Over 
Variable Scale Hierarches of Timespans and Communication Channels. 

A Communication Channel can represent the messaging history of a Class or Subclass of 
Human System Users or Software Agents; it can also capture the Worldlines of Linked 


Spaces in the VR Pane or Context Tags Introduced by the User to aid future recall. 
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A selector widget or 2-D Spatial Chat Room controller can be used to direct utterances to 
specific actors and filter inbound message traffic to view it in full or passively monitor it 


via an Ambient Display. 


In General messages will be depicted as horizontal bars sized proportionately to message 
length on a vertically oriented timeline. Cursoring over a bar will cause its underlying 


text/data/visualization to be displayed in a Linked View. 


The Command Line Interface Pane 


A text prompt with scroll back is NOT enough, we need Documentation and Context. 


Inform 7 follows the Memex tradition to juxtapose 2 resizable Info-Panes with horizontal 
and vertical tabs running along their top and side edges to allow one to select any of 7 


primary views and one of up to 8 sub-views depending on context. 
In VR we aren’t limited to two tabbed views and can “Tear Off’ as many as we desire! 


In addition to Tabbing, Hyperlinks can be followed to jump between views, while each 


pane’s view selection history can be navigated with Backward and Forward buttons. 


A “Source” pane provides a typographically rich Terminal-Pane for entering Quasi- 
Natural Language commands and entire programs that can be copied over from the docs 


with a single click. 


“Documentation” and “Index” panes offers access to two manuals and a sub-catorized 


data dictionary including Maps of how elements are connected. 


Viewspecs 


For Our Purposes, Viewspecs are essentially functions that define how data will filtered, 
styled, projected, and interacted with in a visualization. 

We Build Up Viewspecs by mapping object properties to dimensions and specifying the 

appearance and content of the point at their intersection using an unambiguous subset of 


English that can be defined with a Parsing Expression Grammar 

As a result, Viewspecs are Quasi-Natural-Language Expressions that can be: 
- Named and parametrized and composed from simpler Viewspecs 

- Shared in Email or Visual Meta 

- Derived and Extracted from Live Views 


- Represented with branch-able Wordlines in The Transcript Pane 
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- Modified in real time via Direct Manipulation of the View they represent or via Textual 


directives 


What Can We Specify with Viewspecs? 


We can use Viewspecs to Scale different ranges along a given axis to compress or expand 


the space between displayed points in a view analogous to using Glue in typesetting. 


We can specify how to represent N-Dimensional Objects where multiple items may be 


mapped to the same point in any given 3-Dimensions by linking a series of Views/Spaces. 


- This can take the form of Semantic Zooming (i.e. Replacing) the Original View; 


Rotating the Dimensions of a ZigZag View; Embedding a New View as a 3-D “Cell” in the 


Current View; or Unfolding a selected point into a linked independently positionable 


adjacent 3-D child view that can Persist as we move a Probe in the parent space to select new 


result sets, indicating their size with sound or by altering the Probe’s shape, brightness or 


color. 


Can also invoke Mutli-Dimensional Scalling & Numerical Taxonomy methods to organize 
our data. 

We might also overlay scaled avatars to see who else is present and what they are looking 
at. 

We can enable editing an item’s extrinsic properties by grabbing and dragging it within a 


view. 


Examples of Driving Complex Visualizations with a Command Line Viewspec 


Domain Specific Language (DSL) 


Plot “Start Date”, “Headcount”, and “Number of Milestones” in “Spring Projects” colored 
from green to blue by “Urgency”; embedding each project’s “Project Type” icon in a 10 
by 30 by 20 voxel cell, that semantically zooms into Project Financial Summary Space. 
Unfold the selection’s “Manager” exposing “Experience Level”, “Number of Active 
Projects”, and “Failure to Success Ratio”, lit by relative number of “Complaints on File” 
and colored red if “Human-Resources Flag” is set, or blue otherwise. Set “Human- 
Resources Flag” if “Complaints on File” is > 3. 

Define “Project Financial Summary Space” as a horizontally split planar view of a 


“Project Names” list, bound to a “Financial Summary” outline of the selected project’s 
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subtasks; where the “Financial Summary” outline lists the “Funds Spent’, “Funds 
Projected to be Spent” and “Contingency Fund Balance” of each entry over a linear plot of 


all projects (viewed by budget on a log scale) highlighting the currently selected project. 
¢ Cluster “Butterfly Survey Expedition Specimens” in a 3-D Space based on all of their 


attributes. 


UI Support for Discovery of the Viewspec DSL 


¢ Interactive Textual Dialog — a dialog based “wizard” 

¢ IntelliSense (i.e. code completion hints) 

¢ A Tile-Based Structure Editor (e.g. Blockly) 

¢ A Data Flow / Wiring-Based Editor (e.g. Nodes) 

¢ The Cut, Paste, and Edit of Hyperlinked Examples (e.g. Inform 7) 


The Gestalt We Are Aiming At 


¢ We Want Our Interface to Support Fluidly Shifting Among Multiple Views 
¢ We Want to tame High-Dimensional Data Sets 


* We Want An End User to Be Able to Create New Views On The Fly without needing to 
Hire a Programmer to build a dedicated “App for that’. 


¢ We Want to Leverage Direct Manipulation and Text as Co-Equal Input Channels 


* Once we learn the Viewspec language we can use it to produce effects that would entail 


too many direct manipulation GUI interactions to be worth the effort. 


In a World of fully immersive VR, Old Fashioned Text is the Secret Sauce for getting Serious 
Things Done. 


Bibliography 
¢ Blockly: https://developers.google.com/blockly/ 
¢ Chat Circles: http://alumni.media.mit.edu/~fviegas/chat_circles.pdf 


* Cyberspace: First Steps: https://mitpress.mit.edu/9780262023276/cyberspace/ 


¢ Inform 7: https://ganelson.github.io/inform-website/ 
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Pol Baladas 


There are two great points to be shared after our practical 


explorations: 


¢ Playing with an AI on a spatial canvas: Text is one of the most effective ways to transform 
our thoughts into a physical memory. We can visualize our mental processes, reflect on 
them and even rearrange them spatially to make connections between our ideas or separate 
different concepts. In addition, by extending our thinking processes externally, others can 
join our shared space and help us to reflect and move on with our thinking. Many modern 


tools allow us to visualize our collaborators and co-create by sharing the same space. 


The next question comes when we imagine how to collaborate with an artificial intelligent 
agent in a shared space. What happens when an AI agent can respond across a shared spatial 
canvas rather than only continue what we are writing in one direction? We may be able to 
ask an AI agent to combine different thoughts filling our empty canvas with some ideas to 


help us overcome our creative block. 


Then, we'll become curators of Al-produced content, rather than focusing on the creation 
itself. We can imagine ourselves providing possible directions and letting these AI tools be in 
charge of transforming, organizing, and making connections between our ideas. In that future, 


we become the conductors of an orchestra of agents that write following our orders. 


¢ Discovering new fundamental operations on text with LLMs: When we analyze a tool like 
Fermat under Engelbart's H-LAM/T System one stops at the "M" (Methodology) and 
wonders. I always refer to the handwritten long-division algorithm used at schools to 
explain the "M" in the acronym - it's a good example, for it shows how mathematical 
notation augments us, how pen & paper augments us, and it definitely needs some training 
to use, completing the system under the H-LAM’T lens. In our exploration (using Fermat) 
we can very cheaply play with Large Language Models (LLMs) and, in doing so, create 
complex prompt engineering or specific tasks and abstract them away in atomic UI 
elements like buttons: one for summarizing a text, another that generates counter- 
arguments from a statement, or propose creative solutions for a problem. After imbuing 


our digital workspaces with these Al-enabled buttons, the user starts using them as new 
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fundamental operations on text. Where one previously would cut & paste or find & 
replace, now the user can summarize or criticize a text automatically, extract relevant 
keywords, generate counter-arguments, generate more ideas... in less than a second, 
which makes these (complex) actions feel like automatisms - in other words: new 
Methodologies for working with digital text under the lens of Engelbart's H-LAM/T. 
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Sam Brooker 


Supplementary Material: Devaluing the Work and Elevating the 
Worker 


Early hypertext scholarship recognised the power of the book as both object and artefact, its 
physical stability and defined boundaries asserting the personhood of the writer. “Each author 
produces something unique and identifiable as property” as George Landow (Delany & 
Landow, 1994) put it, while Espen Aarseth described boundaries between literary works as a 
cultural construct, a product of print media (Landow & Landow, 1994). Whatever the 
disadvantages of the print work, it at least ensured that there was an object to be traded. 

The value chain for publishing has historically been comparatively stable, if not always 
advantageous to the individual writer seeking to make a living. Releasing one-off works has 
always required the cultivation of a complex network of engaged, motivated individuals keen 
to access them. This structure of distribution is very labour-intensive, however, and reliant on 
a willingness to wait (sometimes for years) for a work to be released. An advantage of the 
historical publishing value chain is that publishers themselves can (at least in principle) apply 
the tools of marketing and promotion to maintain audiences and build anticipation prior to 
release. They can also (again, in principle) sustain a writer during periods of development and 
research. 

Web-oriented models of distribution offer a compelling answer to some historical 
problems of distribution. The infrastructure it provides promises that works can reach an 
audience without the material costs of a traditional publisher — see its value for writers of 
Twine fiction, for example, who were able to easily distribute their works online. 

Two related questions emerge from this new publishing environment: how best to 
maintain those value-adding networks necessary for commercial success, and how to ensure 
that creative people can make a living from them. “A very small number of authors gained 
visibility during this period”, wrote merritt kopas (Kopas, 2015) of the Twine community’s 
most active period, “and almost all of them still struggle with material insecurity.” Alison 
Harvey too notes that Twine’s emancipatory potential, its much-discussed facility to amplify 
marginalised voices, does not provide “an adequate answer to the problem of creating a 


sustainable life for these game makers.” 


“Individuals could now no longer count on the support of their employers,” wrote Fred 
Turner (Turner, 2010), of the world envisioned by Silicon Valley in the 1990s. “They would 
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instead have to become entrepreneurs.” In the convergent, Transmedia space of the web some 
might argue that the tools for promotion and commercial success are already in place. An 
interlocking complex of platforms — Twitter, Patreon, YouTube, Twitch — provides the 
creative individual with the necessary tools to create and sustain a bespoke value chain which 
can support their working life. If Transmedia is “a collection of different segments of content 
that are brought together into a whole larger than any individual segment” (Evans, 2011) then 
this may offer the creator a means to cultivate such a following. Comedian Brian Limond 
found initial success through a largely self-produced comedy series on BBC Scotland, but 


now dedicates his time to the community he has cultivated on Twitch. 


Ultimately this pattern may invert the standard order of an audience’s relation with 
writers and their work. Rather than curiosity about the person being prompted by 
interest in their output, the relationship is with the person themselves. Works become 
manifestations of an identity or personality which audiences recognise through various social 
channels. By exposure audiences increasingly seem to engage more readily with social media 
content than the ostensible “work” of the individual. Young creatives shift their self- 
identification of employment from medium — filmmaker, writer — to either oblique 
descriptions of personality or medium-agnostic terms like content creator. 

Academic communities have been slow to respond to this change. An artefact-oriented 
media culture that fetishizes objects still prevails in many critical environments - even as 
wider cultural attitudes shift from an emphasis on the work to the persona of the worker. Are 
we really experiencing a fuller inversion of text and context, a perhaps natural evolution of 
poststructuralist ideas? Do audiences increasingly see the creator as locus of enquiry, their 
output valuable mostly as explanatory of person or endorsed as a means of supporting what 
amounts to a favoured stranger’s hobby? What might happen if we move more fully from a 


text-oriented culture to a context-oriented one? 


What does this mean for the writer? There are certainly advantages to making the 
worker, not the work, the locus of attention. An intimate, personal relationship with the 
creator, one that fosters a small but highly motivated audience, may sustain work that would 
be unlikely to find wider appeal. The need to sell thousands of units becomes irrelevant when 
the wider infrastructure costs are so comparatively low. Value networks that do not scale may 


work at the level of the indie content creator. 


And of course, there are disadvantages. Publishers, recognising this phenomenon and 
under significant financial pressure themselves, increasingly expect writers to bring the 
audience as well as the work. Time-poor individuals with limited resources, already 
struggling to complete creative work, now must maintain a complex, coherent identity across 


a plurality of social media platforms. Access to well-funded and well-connected networks 
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will still permit advantaged individuals to get ahead, while those from less privileged 
backgrounds struggle to make themselves heard over the noise. A reorientation away from the 
object may also render the “real” work — the book, the film - a benign hobby tolerated by an 
audience there to engage primarily with the persona of the individual. Building an audience 
imputes an instrumentalist view of social interaction, encouraging performative activity 
which may act against our instinct toward truth and authenticity. Such parasocial 
relationships have consequences for audiences as well, particularly in their deepening 


connection with a creator whom they are unlikely to ever meet. 
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Scott Rettberg 


Cyborg Authorship: Humans Writing with Al 


2022 is AI Spring: new very large language models are exploding exponentially, lending AI 
such as GPT-3 and its various successors and competitors increasing power to replicate and 
respond to human language as active cognizers, producing writing that is becoming nearly 


indistinguishable from human-authored text. 


At the same time text-to-image programs such as DAL-E 2, Midjourney, Stable Diffusion and 
others are enabling new acts of interlocution and image production. At the same time more 
conventional modes of computational narrative systems are being used by electronic 
literature authors in the process of authoring books that engage with algorithms along with 
human-authored language and structures. In this paper I'll explore what these systems mean 
for human authorship and consider examples of how some authors are working with AI in 
creating narratives that are neither solely the product of human consciousness nor AI but the 


product of a hybrid process — cyborg authorship. 
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Yiliu Shen-Burke 


Introducing Softspace 


An initial design for a collaborative spatial knowledge graph. 


Abstract 


A critical step in creative knowledge work is synthesis: the distillation of disjointed data into 
coherent ideas. As information problems become more complex, and good ideas 
increasingly valuable, individuals and groups demand better tools for managing and 
synthesizing knowledge. 

We observe three trends in software aiming to meet this demand: spatial canvases (e.g. 
Muse, Figma), knowledge graphs (e.g. Roam, Obsidian), and collaboration (e.g. Zoom, 
Teams). 

However, legacy tools remain constrained by their flat, bounded interfaces. Our design 
for Softspace proposes a collaborative spatial knowledge graph that transcends the 2D 


paradigm, offering creative knowledge workers the ultimate tool for thought. 
|. Introduction 


Knowledge Synthesis 
The creative knowledge economy runs on great ideas, executed well. 

A critical step in developing these ideas is knowledge synthesis: the work of 
recombining a large, disjointed collection of information into something simple, coherent, 
and valuable. 

A designer synthesizes a wall of references into a beautiful product design. An 
entrepreneur synthesizes stacks of market data into a bold business strategy. A researcher 


synthesizes a myriad of observations about the world into an elegant explanation. 


Synthesis is hard. To do it well, we have to hold many pieces of the puzzle in our head 
at once, and test innumerable combinations of ideas. Synthesis gets exponentially harder as 


the quantity and complexity of the information we’re working with increases. 


Spatial Computing 
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By making the sharing, processing, and storage of information fast, cheap, and reliable, 
computers have become invaluable to knowledge workers. Yet when we use computers for 


knowledge synthesis, it becomes apparent that the UI is now the bottleneck. 


Simplistically: laptop screens are too small and flat for working on big knowledge 
problems. They don’t let us see enough of the puzzle at once, and the way they show 
information makes remembering or reasoning about it difficult when it’s out of view—which 
is most of the time. 

Spatial computing offers a solution. XR headsets let us see, remember, and think about 
far more information than before, by displaying it in immersive 3D. This may be a new 
paradigm for software, but it’s one to which our brains and bodies are exceedingly well- 


adapted. 


Softspace 


Softspace is an XR productivity and creativity app that gives creative knowledge workers a 


powerful new way to organize, develop, and communicate great ideas. 


In our design, users work with conventional content types (e.g. text, images, PDFs, and 


websites) within a radical new paradigm: a collaborative 3D knowledge graph. 


Through a workflow that combines elements of notetaking, mindmapping, and 
moodboarding, users build up spatial information workspaces that reflect the structure of 


their ideas. 


Users then compose and export linear syntheses of their knowledge graphs, in the form 


of markdown files, for use in downstream workflows. 


The immersive virtual workspace allows collaborators to step into the same information 


space for discussion and co-creation, regardless of physical distance. 


Il. Design 


Items 


The entity primitive in Softspace is the item. Items are single pieces of content data, or 
containers that hold other items. Items correspond to what are called blocks in some other 


knowledge-management tools, such as Notion or Roam. 


Content Items 


Conceptually, content items map best onto single files, although in cases such as text and 
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URLs this correspondence does not strictly hold. Initially, content item types in Softspace 


include: 

* text paragraphs 

* static images 

¢ PDF documents 

* bookmarked websites 

Future updates to Softspace may implement, among others: 
* tweets 

* videos 


* podcasts 


Container Items 


As their name suggests, container items hold other items. The initial design for Softspace 


only specifies a single type of container: the topic. 
Topics can either be expanded or collapsed. 


¢ While expanded, the contents of a topic are displayed in a fixed ordinospatial layout that 
makes use of all three spatial dimensions while constraining contents to a single linear 
order. This ordinality makes topics mappable to conventional document formats (such as 


markdown), and usable as 3D notes. 
¢ While collapsed, the text contents of a topic become hidden. However, backlinks from 
hidden text to other topics are visible as connections between the collapsed topic and 
referent topics. Images, PDFs, websites, and other non-text contents remain visible—they 
float free of the collapsed topic, but remain visually and spatially connected to it. 
Topics have a fit/e: a string value that identifies that topic within the workspace. Future 
updates to Softspace will implement topic aliases: alternative titles (to accommodate 
capitalization, synonyms, orthographical variants, etc.) by which one topic is identified. 
When exporting a Softspace workspace, each topic is interpreted as a markdown file 
whose filename is the topic title, and whose contents are those of the topic. 
For performance and technical implementation reasons, the initial design of Softspace 
will only permit a single topic to be expanded at a time. Later updates will allow multiple 


topics to be expanded simultaneously. 


For performance and technical implementation reasons, the initial design of Softspace 
will not permit nested topics—i.e. topics cannot contain each other. Later updates will enable 
this. 
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Transclusion 


Transclusion is a term coined by Ted Nelson in his 1980 publication Literary Machines. It 
refers to the concept of including a single piece of content across multiple contexts as live 


instances, so that a change in one instance is reflected across all instances. 


Softspace implements transclusion by allowing item types to be contained by any 
number of topics. Adding a transcludable item to a topic does not remove it from its other 
containers. 

Initially, this will be possible for item types which are highly atomic (semantically 
independent of their immediate context) and/or relatively immutable. Text items do not meet 


this criterion, and can therefore only be contained in a single topic at a time. 


Transclusion allows topic containers to function as tags. 


Backlinks 

The popularity of the note-taking app Roam Research (and Roam-like apps) can be largely 
attributed to its use of the backlink as a core interaction primitive. A backlink is an in-line 
reference from text to a conceptual entity. In the case of Roam, backlinks point to notes. In 
Softspace, backlinks point to topics. 

Our design borrows the [[]]-notation of Roam. Terms within a text item which are 
surrounded by double square brackets will be visually and spatially linked to the topic with 
the same title as the enclosed text. If no such topic exists, one will be created. 

When a topic is collapsed, the backlinks from its hidden text items remain visible as 
indicator lines that connect the collapsed topic to referent topics. In future updates, these 


indicators will display the snippet of text which contains the backlink. 


Spatiality 
Whiteboarding apps like Miro have proven the tangible value of being able to lay out 
information spatially in a software tool. Many apps which feature a canvas for UI design 
purposes, such as Figma, are often used as general-purpose boards instead. New tools built on 
this basic pattern seem to emerge daily, from Muse to Heptabase to Apple’s upcoming 
Freeform. 
Spatial interfaces are effective because our brains have evolved to be astoundingly good at 
perceiving, remembering, and interpreting where objects are in our environment. 


But a spatial canvas displayed on a laptop screen suffers from three drawbacks: 
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1. 2D supports less spatial complexity than 3D, limiting spatial semantics. 
2. The view cuts off at the edge of the screen, limiting contextual awareness. 


3. The user is not situated within the workspace, limiting spatial memory. 


The design premise of Softspace is a 3D spatial canvas within which the user is situated; 


therefore, it bypasses the above three constraints. 


Further, our design incorporates three distinct modes for the spatial positioning of items 


in a workspace. These modes are optimized for different phases in the workflow. 


Ordinospatial Layout 


Within an expanded topic, items are laid out using a front-to-back, left-to-right, top-to-bottom 
system called an ordinospatial layout. All three spatial dimensions are used to arrange 


contents, but there is a definite order that makes each topic interpretable as a linear document. 


This mode is best for the content a user is working on directly at that moment. Using 


this layout is conceptually similar to drafting a 3D note. 


Force-Directed Layout 


Items whose positions are not fixed within an expanded topic are subject to the force-directed 
layout system. This is a simulation-based layout system that automatically gathers related 


items closer together in space, and pushes unrelated ones further apart. 


Related items are those with a semantic relation in the knowledge graph. The initial design 


specifies two such relation types: 
* containment: the relation between a topic and its included contents 


* reference: the relation between a text item and topics it links to via [[]]-notation 


This mode is best for content that is not currently being worked on. Items move themselves 


into a spatial configuration that makes visible the relationships between them. 


Cartesian Layout 


Items not in an expanded topic can also be pinned in place, so that their position and rotation 
is no longer determined by the force-directed graph simulation. Instead, they remain at a 


fixed, user-determined Cartesian coordinate. 


This mode is best for reference items that should be held in a specific spatial configuration. 


497 


Workspaces 
The highest-level organizational unit of Softspace is the workspace. 

Items are always created within a workspace. Initially, items will only have a single 
parent workspace; later updates may enable cross-workspace transclusion. 

Users can create, manage, open, close, and delete workspaces using an in-headset UI. 
The initial design for Softspace only permits a single workspace to be open at a time. 

Workspaces can either be /ocal or cloud. Local workspace data is stored completely on- 
device, and are not multiuser-compatible. Cloud workspaces store their data to the cloud, and 
are multiuser-compatible. Softspace will launch with only local workspaces; cloud 
workspaces will be implemented shortly afterward. 

User permissions are managed at the workspace level. Permissions roles can include 


owner, administrator, and guest access. 


Workflow Integration 


A key challenge when using mobile devices for productivity is the lack of a common file 
system. XR headsets are no exception. This deficit adds friction to the process of bringing 
files into and out of mobile software. If this friction is too high, it can feel like work gets 


“stuck” in the device, which understandably deters use. 
Softspace is a designed to minimize this friction. It does this by: 


* Prioritizing support for content formats that are common across knowledge workflows, 


such as images, PDFs, and markdown files 


¢ Integrating with popular cloud file storage services like Dropbox, with automatic exports 
to maintain a readily-accessible copy of work outside the headset 


° Implementing a full-featured in-app browser that makes the web easily accessible from 
within the headset 


The goal is to maximize upstream and downstream compatibility with existing workflows, 
while retaining the unique advantages of this new computing medium. Users can quickly 
bring files into Softspace, work on them there, then easily access the contents of that 


workspace from their other devices at any time. 


Common File Formats 

Initially, users will be able to import: 
¢ Text files (.txt) 

¢ Markdown files (.md) 
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* Image files (.jpg/.jpeg, .png, .tif/.tiff) 
¢ PDF files (.pdf) 


Users will be able to export workspaces as a collection of 1) markdown files that correspond 


to its topics, and 2) image and PDF files. 


Cloud Storage Integration 


Cloud file storage access is available in the form of a simple 2D web app that is accessible in- 
headset. It allows users to log into their cloud storage accounts, select files and folders for 


import, and select folders to export workspaces into. 


The first such integration will be with Dropbox, because of its large user base. We are 


also exploring Google Drive and local network drive access. 


In-App Web Browser 


A good web browser is central to almost all knowledge workflows. We implement a full- 
featured browser within Softspace that gives users access to the rest of the web, including the 


web app versions of complementary tools. 

The in-app browser will allow the user to: 

¢ Browse websites, including web apps 

¢ Bookmark websites for future reference 

¢ Save images files from the web into workspaces 

¢ Snip any portion of the browser window into workspaces 


* Copy/paste text to and from text items 


Multiuser Support 
Although by catalyzed the pandemic, we expect the importance of remote work to continue 
well into the future as companies seek top talent, and talent seeks geo-flexibility. 

However, video-based remote collaboration tools fall far short of the creative magic 
that is possible when working together in-person. XR closes the gap by creating a true sense 
of social co-presence between collaborators in the same virtual workspace. 

Softspace will not initially support multiple users, but its technical architecture has been 
designed from the beginning with multiuser collaboration in mind. We will enable this feature 


once cloud workspaces are implemented, as this is a key technical prerequisite. 


Interaction Model 
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Spatial computing remains in its infancy, and is evolving rapidly and divergently. 
The Softspace interaction model is designed to rely as little as possible on the specific 
features of today’s headsets, and to be highly portable across hardware and input paradigms. 


Therefore, our design only assumes a head-mounted 6DoF AR device (passthrough or 


see-through) with high-fidelity hand tracking and bluetooth keyboard support. 


Augmented Reality 

Full-occlusion virtual reality is unsuitable (at least as the default mode) for a tool intended for 
use in professional settings. Blindness to one’s immediate physical surroundings gives rise to 
a sense of unease and vulnerability. This prevents many users from entering the state of flow 
that is necessary for doing their best creative work. 

The design of Softspace is premised on an augmented reality paradigm that allows 
users to see their immediate environment. Virtual UI elements appear to float in this space. 
Initially, it will not be possible to anchor items to specific points in the physical environment, 
but this functionality will come with later updates. 

Happily, even the low-resolution passthrough augmented reality of the Quest headset is 


sufficient to dispel the discomfort that arises from visual occlusion. 


Hand Tracking 


Currently, the most common input device for XR headsets is the hand controller. Optimized 


for gaming, this device is poorly suited to productivity use cases, because: 
¢ It must be held in the hand at all times, precluding the use of a keyboard 
¢ Its form, balance, and button placement are reminiscent of weaponry 

¢ It adds two more devices to keep charged, remember to pack, etc. 


Therefore, all non-keyboard inputs in Softspace rely only on computer vision-based hand 


tracking, which has already been developed to a very high level of usability and reliability. 


Locomotion 
The way that virtual objects are overlaid on the view of the physical environment in 
Softspace makes user locomotion through the workspace technically equivalent to the spatial 
repositioning of the workspace around the user. The only difference between the two is frame 
of reference. 

Locomotion—or correspondingly, workspace repositioning—is initiated by forming a 


fist with one or both hands. This action “grabs” the workspace where the hand(s) is 
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positioned. The user then moves the grabbing hand(s) to move, rotate, and scale the 


workspace. 


Manipulation 


Our philosophy for the design of the Softspace UI can be characterized by leveraged direct 
manipulation. We want to give users a feeling of high agency, effectiveness, and control 
when interacting with objects in the workspace. 


Users highlight UI elements with a line-of-sight system, which uses an imaginary ray 
from the eye to the hand as its targeting vector. 

Once an item is highlighted, pinching the tip of the index finger to the thumb is 
interpreted as a click action, which either causes the highlighted object to be used or grabbed. 


Movements in the grabbing hand cause the grabbed item to be repositioned, with a 


leverage factor being applied to its motion along the user’s vector of view. 


Grabbing an item with two hands enables rescaling and resizing. 


Text Input 


Given the centrality of natural language in knowledge work, text is a first-class content type 


in Softspace. Fast, accurate, frictionless text input is therefore absolutely critical. 


Given these requirements, there is simply no viable alternative to the use of a physical 
(bluetooth) keyboard as the primary text input device. However, a backup virtual keyboard is 


available at all times. 


Future releases of Softspace will explore additional text input methods, such as speech- 
to-text. 


Art Design 


The designer of XR software has much more control over the sensory input of their user than 
the designer of a 2D app does. While in Softspace, everything a user sees (and much of what 
they hear) is the result of decisions we will have made. This gives us great power to shape the 


user experience, but also comes with greater responsibility to ensure it’s a good one. 


Two principles underpin the art design of Softspace: comfort and productivity. 


Comfort 
Comfort in Softspace has both an aesthetic and a performance component. 


Given the full-immersion nature of XR, less is more. Our aesthetic design is restrained 
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and minimal. We rely on a limited palette of colors and a familiar set of geometries. We 


prefer to subtract, adding only when necessary. 


Performance-wise, maintaining a smooth 90fps on the Quest 2 (and 72fps on the Quest) 
is critical for user comfort. We therefore make creative use of a few simple meshes and 


shaders, to minimize compute load as the number of items in the workspace increases. 


Productivity 


Most XR software today is for gaming, entertainment, or other use cases where there is 


heightened value in sensory stimulation and excitement. 


In contrast, we are building software for deep work. To support this, we wish to 


promote focus, creativity, and flow instead. 


Softspace UI elements, passthrough color filtering, animation behaviors and velocities, and 


other aspects of its look-and-feel are calibrated to foster these qualities. 


Ill. User 


The intended user of Softspace is a high-agency knowledge worker whose livelihood depends 
on her ability to quickly and effectively synthesize complex sets of information. She might 
be: 


¢ An entrepreneur writing a product requirement doc 

¢ A design researcher summarizing user interviews 

¢ An independent analyst drafting a Substack post 

¢ An architect crafting a deck about a new project 

¢ A grad student outlining a chapter of her thesis 

She currently uses a combination of analog tools (pen and paper, physical boards) and digital 
ones (Notion, Apple Notes) to collect information, make sense of it, and draft documents. 
Crucially, she feels an acute frustration with the limitation of existing tools. She may be 
exploring spatial canvases (Muse, Figma) or structured note-taking apps (Roam, Notion) to 
help her manage and make use of her knowledge base. 

These apps are steps in the right direction, but she wonders why there still isn’t software that 
gets close to the creative magic of a shared team project space. 

Of course, to be able to download and use Softspace, she will need to have access to a 


compatible headset (e.g. a Quest 2). 
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IV. Flow 


Workflow Phases 

Formally, the intended user flow of Softspace can be divided into three phases: 

1. Collection: adding relevant information to a workspace via cloud storage and the web 
2. Construction: building up the knowledge graph by writing notes and composing topics 
3. Collation: composing synthesis topics intended for export as linear outlines or drafts 


In practice, we expected users to cycle through this flow many times, jumping between steps 


as they seek to make sense of a knowledge problem and explore different solutions to it. 


Example Flow 


For example, this is how somebody might use Softspace to draft a design proposal: 


1. Collect images and PDFs related to the project into a Dropbox folder. Export notes 
from a note-taking app as markdown files into the Dropbox folder. This content likely 


includes: 
project brief 
reference images 
client interview notes 
previous project materials 
2. Launch Softspace on headset, and create a new workspace. 


3. Using the cloud storage UI, import the contents of the Dropbox folder to the 
workspace. 


Text and markdown files are converted into Softspace topic items 

Images and PDFs are converted to the corresponding content items 
4. Build up a project knowledge map from these contents by: 

Creating and writing text blocks 

Creating topic and adding content to them 

Referencing various topics from within text 

Adding images using the in-app web browser 


Copy/pasting text using the in-app web browser 
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Bookmarking URLs using the in-app web browser 
5. Create a new synthesis topic which is intended for export. Compose a draft of the 
design proposal through a combination of 1) including content items already in the 
workspace and 2) writing new text that ties ideas and content together. 
6. Using the cloud storage UI, select a Dropbox folder to export the workspace to. (This 
can be a one-off action, or be set to recur automatically.) 
7. Copy the synthesis topic’s markdown file into a word processor for editing and 


formatting. 
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Yiliu Shen-Burke 


Journal Guest Presentation : Discussing Softspace 


https://youtu.be/SjQrimm4mGU 


Yiliu Shen-Burke: My own journey into the specific question of VR and text is actually 
a little bit circuitous, it's not where I started out. In fact, only very recently in my process 
of prototyping and building what is ultimately intended to be a commercial product, an app 
that people will download and use, and be financially self-sustaining. The role of text has 
only recently become a very central one, and probably, honestly, the reason for that was 
because I was a little bit scared of this question, of working with text in VR. Because I think 
all of us, who use computers on a regular basis, we know how central text is, and it seems to 
be the point at which all of the shortcomings of the hardware, up until very recently, came to 
the most obvious head, where it seemed like the screen resolution, more than the screen 
resolution, the fact that text is just presented in it on a 2D plane. And it didn't seem to be a 
need to work with it in a three-dimensional way, or at least at first glance. I think all of these 
factors contribute to my reticence in tackling text. I recently I started doing it, and it's been 
absolutely, incredibly mind-opening, and I think there are, to refer to earlier about the 
combination of spatial interfaces within text being one of the most transformative 
technological opportunities that are coming our way, I totally agree with. And what Brandel 
was saying about this combination actually being an existing way back in the earliest 
histories of computing. And also with Ivan Sutherland’s AR Experiments. Actually, people 
have been wanting to do this for decades, and only very recently because of these billion- 
dollar tech giants and their investments, it has become technically usable for people like me 


just to throw things together, and actually, have it work and be usable. 


Five years ago I was a VR research resident at an artist's studio here in Berlin. I 
was on leave from architecture school. There's an artist here in Berlin called Olafur Eliasson 
who has a very big operation, over 100 programmers, designers, craftspeople, PR people, 
social media people, and among them myself, the VR research resident. And the reason why I 
got this job was because the DK2 had just come out, someone in the studio had bought two of 
them, and they realized that there was nothing to run on these apps. They couldn't just 
download something and run on it, was the very early days. So I convinced someone that I 
knew enough programming to do something interesting with these headsets. I didn't. But I 


was able to, very luckily, throw together enough of something every two weeks or so to show 


505 


the rest of the studio that I was able to teach myself more and more of these, sort of, technical 
skills required. But coming from architecture, I was very comfortable with remodeling in 
Unity and things like this. And what happened over the course of that one year was that, we 
were building these prototypes to explore the potential of virtual reality as either a medium 
for actual artworks, or as a tool for the production process of other works which may not end 
up being digital at all, but as a working environment. Both very, sort of, fruitful and still very 
fruitful avenues for the application of the technology. At the time I was using Evernote, 
because that was kind of like the best digital notebook that was around, or the only one I 
knew about, and very quickly, it became super annoying to use Evernote to manage those 
processing cycles. So I was trying to build something every two weeks, collecting ideas by 
talking to people in the studio, throwing together a working program, and then, showing to 
them at the end of the cycle. And to manage all the ideas that were coming in, and all the 
ideas were coming going out, I was using Evernote, and it was a huge pain to make sense of 
all the ideas that I’ve been collecting, and there's stuff in there that you'll never find again. It's 
a classic issue with 2D UIs. And at the same time, I was in this artist studio that is just 
covered from floor to ceiling, wall-to-wall, taking up this entire old beer brewery in Berlin, 
with the physical artifacts of the design processes, and creative processes of the other teams 
around. And so I started thinking, and this isn't just me, there were also snippets of this idea 
during the demos and during my conversations with artists themselves, about why we 
couldn't use this inherently spatial medium to actually start recreating some of the 
advantages of a physical workspace with physical artifacts, and pieces of information 
around. And this really started getting me thinking, and thinking, and I started prototyping 
some things, pulling information from their in-house CMS that used to power the website 
with archival images and things from previous exhibitions and works. And it quickly became 
clear that, "Okay, an artist studio is not the right, kind of, environment to start trying to build 
a brand new general-purpose tool." And I was coming from architecture school, I knew I 
didn't actually want to become an artist or an architect for that matter. And so I decided, 
"Okay, this is probably a tech start-up. I don't know if that is, but that sounds a thing, and 
people give me money to do it." 

So in 2018, we founded a company, got a little bit of investment from an accelerator 
in Silicon Valley, and started trying to build a spatial virtual studio for designers called 
Softspace, and that's still the name of the company and the product. And at the time, the 
overriding, I guess, was quite skeuomorphic. A paradigm that I had was that you would have 
this very large like, your dream studio workspace, where the laws of physics, and the 
constraints of physical materials didn't apply, but you would still have a lot of the behaviours 


and affordances of the physical media that, especially, visual artists or designers are used to 
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working with. 

I’m going to just quickly jump into my headset now, Frode, and show you the version 
of the software that we built and released on SideQuest, which is an indie app store and it's 
where we first tested our vision for what the future of creative knowledge work might be. 
SoftSpace is what we called it, just for shorthand SoftSpace version 2020, because it was 
first released in July 2020, almost exactly two years ago. And this is what it looks like. I 
know the field of view here is not going to be massive, so I’m trying to keep my hands in the 
middle of my field of view. But you can see here that what you have is, very literally, a 
massive white wall room. So what I’m doing now is, I’m actually sharing the screen instead 
of the feed now, so that should work. It forces Zoom to just show this through-the-lens view 
to everybody. The only downside to this mode is that the frame rate is going to be quite low, 
so I’m just going to move very slowly, and that way, you should see what I am doing. Does 
that sound good? Okay. So I’ll just (indistinct) again which is that SoftSpace version 2020 is: 
¢ Inmany ways, a recreation of the big, empty, white room that you can pin stuff up in 

wherever you like that came out of the experiences I have working at this artist's studio. 
That's a very simplistic way of describing the paradigm here. 
¢ Aslightly secondary, or second-order understanding or explanation for why this app 
works the way it does is that, I didn't have any idea of what the right layout paradigms, or 
what the right revealing structural relations between pieces of information in a three- 
dimensional user interface should be, at all. 
This version of the app was never intended for a broader distribution on an official app store 
or anything. It's what we were calling internally, a cartesian sandbox. You can grab any item, 
and I just wanted to do something over here to find my text, so I’m going to grab this little 
text note here, so I can make it big, I can make it small, I can move it freely in a three- 
dimensional space, and I can rotate it within a certain, like, it snaps in terms of its rotation, 
but I can, kind of, put it wherever I want. And within this cartesian playground, in a sense, 
you have all of the freedom and power of spatiality to express relations between things, by 
positioning them and scaling them relative to each other. So if any of you are familiar with 
the whiteboarding app, Miro, this is essentially a 3D version of Miro. And you can see that 
there are objects which are containers, and these containers can hold other things. There are 
objects which are boards, which are 2D containers, you can pop items off and put them onto 
the boards. And there even is, for example, a fully functional built-in web browser that you 
can agree to get cookies installed onto, and you can Google for topics that might be of 
interest during your research process. And if you find an image of something you like, you 
can pop it out and save it to your workspace, like so. 


Frode Hegland: So one of the things we have discussed, and that I have a great fear of, is 
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ghettoizing or sandboxing, or whatever it might be. And currently, there are, of course, 
many applications where you can view your laptop screen in VR, and they're all quite neat. 
But the problem is that, that is just a texture, it's just isolated. So can you, please, elaborate on 
how you took something from that screen into the room? Because, I think, that is really 


wonderful and important. 

Stephanie Strickland: Can I ask a question too? Which is, can you get the text off a 2D 
surface and set the letters free in the 3d space? 

Yiliu Shen-Burke: Not yet, but I will get there. So that's a really great question. 

And Frode, the context of the question you asked about sandboxing is exactly what 
prompted me at the end of the 12 months of having this version of the app out in the wild, 
hearing feedback from that, watching people using it, deciding, "No. This is not the 
direction." And kickstarting my latest cycle of prototyping. 

But just to really speak, specifically about the technical, I guess, implementation of this 
popping things off, this is a fully functional Firefox web browser running inside the 
SoftSpace app. And I’m using a package called Viewplex, and this is all running in Unity 
C#. But what Viewplex lets me do is pass messages between the HTML-JavaScript 
environment inside the browser. And the C# environment outside of the browser in this app 
itself. And so, whenever my cursor touches something that is marked as an image, the 
browser tells the VR app, and the VR app then knows, "Ah, okay. Your cursor is actually 
hovering over something that can be downloaded out as an image." And so, you see this icon, 
it changes from this manipulation icon to this copy-out icon. And if I press the trigger then, 
actually, it's going to the URL, the original URL of this image, and downloading the full 
resolution version of this image, and then saving it to the cloud storage backend that we're 
using for this particular version of the app. 

Frode Hegland: Can you also copy and paste that way? 

Yiliu Shen-Burke: You can also copy and paste, and the way you do that is, there's a 
separate tool called Snip. For example, if this whole thing, this website were not actually 
accessible as an image, sometimes website makers decide to hide everything under an 
element that blocks your ability to directly select things, I can press the trigger on this 
controller, switch to a snipping tool, which will just make a copy of this entire, any part of the 
web texture that I wish to make a copy of. So that's just a straight, one-to-one texture copy of 
what the browser was seeing. And that same tool, actually, just with a click, lets me make a 
copy. 

Frode Hegland: Can you have more than one person in this room, currently? 

Yiliu Shen-Burke: Yes. This is fully multi-user enabled. Okay, as I was saying earlier, this 


was released on SideQuest, in the app store, which is a side-loading early access marketplace 
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for apps. We have gotten 3.000 registered users, and 6,000 downloads of the app, and even a 
handful of paying users of the premium plan. But when I was thinking about, “Okay, how 
would we, then, take this and bring it to market, to the official app store for the Quest 
headset?” Which is a much larger user base, I realized that, I was, at a fundamental level, 
immensely dissatisfied with the paradigm that this app represents. And in fact, this paradigm 
was never intended to be the final paradigm. This was very much research, an applied 
research vehicle, to understand what people wanted to do if they were given the ability to 
place 2D content, largely because, we're talking about images and text blocks, what they 
would want to do with 2D content in a 3D space that they could freely place things in, and 
move themselves around? And this is what prompted, then, me to kickstart the cycle of rapid 
prototyping, and then, releasing those prototypes to the public, on Twitter, mostly. I started in 
February of this year, which has been incredibly rewarding and fruitful, and I’m just going to 
show you a couple of those experiments which will then lead you all the way up to the final 
point of where I am today, questions I have for you as a group of people who have been 
thinking about these problems for a lot longer than I have, or much more deeply than I have. 

And also, I believe it was Stephanie, who asked the question about being able to pull 
text off. Pll get to that at the very end of it. I don't want to forget about you, I'll show you 
what I mean. 

Stephanie Strickland: And also changing the font at any point and so on. 

Yiliu Shen-Burke: I see. Okay. Yeah. So specific text formatting tools, I actually have not 
spent a lot of time building out, but I will show you what I have been looking at. So this is, in 
a literal nutshell, if you scale (indistinct). This is SoftSpace version 2020. And a very 
instructive research environment for figuring out what the actual paradigm for SoftSpace that 
would actually tap into the full power of this CDM should be. 

Frode Hegland: Do you need a PC to run this or can it be run independently in just a 
headset? 

Yiliu Shen-Burke: This is all standalone. Although we do have a PC version, this is running 
completely on the Quest. I’m actually going to go backwards a little bit from the prototypes 
that I’ve been building starting from February, and I’m going to start with prototype number 
3, which is the latest one that I’ve published, and which I was very happy with for getting a 
bunch of attention on Twitter. 

Prototype 03, there are a couple of things you will notice right off the bat. The first 
thing is that, this is an augmented reality, or mixed reality enabled application. So 
there's no longer this completely immersive virtual environment that you're thrown into, 
which blocks out your view of your physical surroundings. And I had come to the realization 


that, even me personally, someone who's worked in VR for many years now, and should be 
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very comfortable with this medium, every time I went into VR and covered up my eyes, I had 
this really inevitable low-level sense of unease and vulnerability that came from not being 
able to see my surroundings. And so, when Oculus, back when it was still Oculus, released 
the pass-through SDK and made it possible for app developers to access pass-through video, I 
was sceptical at first, I tried it, and I just thought, “This is the future. SoftSpace will always 
be an augmented reality or a mixed reality app from now on.” And then the second thing 
is that, I just put my controllers down because everything is now being controlled by my 
hands only. So this is just the standard Oculus hand tracking. It's gotten very good. It's gotten 
a lot better than it was when I first started playing with it. And the combination of these two 
features, the pass-through video, and hand tracking, also were the enabling factors for me to 
really start thinking seriously about text input because it means that you can have a physical 
Bluetooth keyboard that you can easily start typing on, without putting your controllers 


down first and that you can see because it's pass-through enabled. 


So this is the context of the prototyping cycle that I currently embarked on. And the 
specific question that Prototype 03 was trying to answer was, how could you map an 
ordinal set of information onto a three-dimensional space? And I’m sure you all know 
what those words mean, but specifically what I mean is, in the previous version of SoftSpace, 
and as Frode was saying, there is this issue that information that is arranged in a three- 
dimensional cartesian layout, 2D has this issue as well, but 3D makes it interesting for me, 
more impossible, is forever stuck inside a 3D environment. 

Because there is no reliable, dependable, sensible way to take a collection of 
information that is arbitrarily placed in a 3D space and, for example, export it to something 
you could email and have someone else read it on their phone on the train. And if that's going 
to be the case, then, I mean, I found this from our users, and from myself as well, you're just 
not going to be willing to invest time and energy into working on something if it's only 
going to be stuck inside the headset. 

And by the way, 95% of the time, you yourself, in the future, will not have access 
to a headset. So if the work you do, you can't even access it yourself reliably, you're just not 
going to want to put any time into it. And so, Prototype 03 was trying to figure out, “Okay. 
Could or in what world would it make any sense, for the underlying data structure that 
your 3D workspace is represented, to actually be an ordinal data format?” For example, 
I’m just using an example here. Markdown. So if everyone's familiar with Markdown, a 
wonderful standard for interchange between different applications. 

Could a VR, AR app actually be working with Markdown? And what would that 


even mean? And so that's the one question. 


And the dumbest possible way, of course, to answer it would be, and this is 
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something that you do see out, around in different applications, would be just to put a 
2D window inside a VR that has a 2D-UI, and shows you a Markdown file. It shows your 
website or whatever. Shows you ordinal content the same way you see it on your laptop. But, 
of course, if you take that approach, you're giving up all of the richness of the expressiveness 
of spatial semantics. You're giving up the whole point of putting the headset on. And I 
didn't want to go to that. 

I’m sure other people are working on the virtual desktops because they're going to be 
amazing, but that's not just where I felt like I can make the biggest contribution. So how 


would you represent the Markdown file in 3D without resorting to importing a 2D paradigm? 


Markdown in Ordinospatial Layout 

The answer I came up with, and I’m going to call here the Ordinospatial Layout, is 
one where you do have a set of very strict rules for determining the order of things. But 
you're using all three spatial dimensions to express that order. 

So to make it clear what I’m talking about. This block of text, this text block, is its own 
object, and it can live at a specific point place in this column. And so, if you were to take this, 
and you were to export it as a Markdown file, which, by the way, the headset is doing, this 
entire workspace is, right now, saved out as a Markdown. Every time you write something 
here, it's saved out as Markdown. This could be the first paragraph, this could be the second, 
this could be the third, etc. You can also move it over here, and maybe, I just want to place 
another block of text below this. And I can start typing on it. I can say, “Hello.” I really can't 
touch type, by the way. “Hello, everybody.” 

And so, you can see that I can move this along this plane. I can start constructing 


columns of text. 


And I can move things relatively freely, if I want to move this thing way only over here, 
I don't want to think about it, for now, I can do so. But it has a definite point and definite 
order within the global set of content: There's no place I can put it that doesn't have a 
meaning. 

And to extend this to the third dimension, I can always move these blocks of text 
between these series of planes that are, actually, not visually represented at all. That's one of 
the shortcomings of this prototype. They're invisible and you, kind of, have to just know that, 
"Okay, there will be a plane there if I pull this thing far enough." So this is prototype number 
three, and this got a lot of attention on the internet, and people started asking, "Could you do 
this? Could you do that? Could you start breaking free of this very strict ordinal layout?" And 
it was great because I suddenly started having really rigorous and in-depth conversations with 


people who had thought very seriously about this kind of question. How do you represent 


S11 


ordinal content in a three-dimensional way without losing the benefits of one or the other, 
including being introduced to this group here? And what I just want to show you now, again, 
I’m jumping through between the prototypes. Stephanie, you asked about the possibility of 
pulling something off. And so, in this particular prototype, you can move things around, but 
the objects are always, or the text blocks are always going to have a place within the 
underlying, essentially, Markdown file. There's no way that I can pull this over here and say, 
"This is going to be in a completely different area altogether. I don't want to think about it." 
It's going to end up in the Markdown file, this entire space is just one 3D Markdown file. 
And, of course, this cannot be where this experiment ends. And so I want to show you 


prototype number four, a work in progress because is what I’m currently working on. 


Prototype 04 

Yiliu Shen-Burke: Okay. Well, I hope you have some capacity left for both enjoying this 
demo, and also, questions, critiques, thoughts, avenues of research, and exploration. Because 
this is exactly the phase I’m in right now. So, okay. Prototype 04 says, "Well, yes, it's great 
that I can take these blocks of text, and I can work with them in 3D, and have this beautiful 
board of stuff in front of me that I know will be exportable easily as an email, or exportable 
as presentation, or I can finish writing here, or I can draft up something here and finish 
writing it on my phone, on the train ride home." But aren't we over-constraining things a little 
bit? What if you want something that is not actually in this document, let's think of this board 
as a document, but it's related to it and lives nearby, or maybe lives in another document over 
here because it's a 3D canvas, so we can do all these things that we can't easily do in a 2D-UI. 
Well, maybe you could just take something, like this text block, and pop it off and just have it 
start floating nearby. And you could run a little 3D force directive graph stimulation, so that 
everything is spaced out very comfortably. Is everyone here familiar with Roam Research, by 
the way? Okay. So using something like Roam Research, it's double square bracket notation 
for indicating references to other topics. Maybe as I’m typing here, I could start adding 
references on this topic or that topic, and those references, actually, become newly created 
free-floating objects, which then start, through the force-directed graph simulation, start 
pulling the blocks of text, which are related to that block you're working on, closer to where 
you are working. And pushing the things that are not so related further out, so you start 
getting a three-dimensional representation of these semantic relations of all the elements 
inside a project you're working on. And so you can see here, I just pulled a couple of little 
blocks of things off, they're floating around, they try to flee from each other, so. 

Stephanie Strickland: I do have a question here. Who decides what's related to what here? 


Yiliu Shen-Burke: You do it manually. 
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Stephanie Strickland: You said that things started aggregating on the basis of their relation. 


Yiliu Shen-Burke: Yes, and so, what I was mentioning earlier was, in Roam, what you can 
do... And this is my current work-in-progress prototype, so it's not implemented here. But, in 
Roam, you could create a tag, or create a new topic, actually, in Roam it's a new note. You 
can create a new note, but using this notation of a double square bracket, and say, for 
example, Frode, right? And this would create, automatically, for example, a Frode object over 
here. And then, in other parts of the document, where I have mentioned Frode, or I’ve written 
exactly this string of text, [[Frode]], the Frode object would start floating, and the thing is, 
there would have to be some visual indication of the relation. So there would there be some 
line that would draw from... 

Stephanie Strickland: Okay, so my question is; that means that ahead of time you had to tag 
things. So what I’m interested in is, what kind of space, whereas I glance over it, and now the 
connections are occurring to me that have not occurred before, am I able to create some kind 
of skeleton, or structure, or whatever kind of structure I use, a tree or otherwise, that I can, on 
the fly, create this thing? I don't want to have to have this (indistinct) ahead of time. If I 
decide ahead of time, I already know all I need to know about the text in it. 

Yiliu Shen-Burke: Yeah, I see what you mean. I would, actually, say that the secret to Roam 
Research, and its success, was its insight. Even if you are manually adding in these references 
as you're writing up your notes, it doesn't mean that you are able to hold in your mind a 
global picture of where else an entire set of notes you've ever mentioned in this one topic. 
And so you're always working locally and you're saying, "Okay, today I’m going to write up 
some reading notes about this book that Frode presented." For example. And I’m reading 
some reading notes I’m going to create some square brackets to annotate topics. And then, 
you switch to a different view in Roam, and you see all the other places where that topic has 
been backlinked, and you have a global picture. And so, I would say that, actually, it's not that 
you have to tag things ahead of time, it's that, as you are working, you're just pulling out 
these instances of references, and then you take a step back, and you look around, and you 
See. 

Stephanie Strickland: Okay, wait a second. All relations are not references. Okay. So you 
have a very citation-based idea. I mean, obviously, a huge database and you can aggregate 
different parts of it as you query it different ways, right? But the point is, how do you create 
different structures of relation on the basis of interacting with lots of texts. Do you see what 
I’m saying it's (indistinct)? 

Yiliu Shen-Burke: I see what you're saying. I actually, so, I don't know if you are familiar 
with Jack Rusher he's based here in Berlin, and he worked on semantic web, and he and I had 


this exact same conversation. And you're absolutely correct. Not all relations between two 
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things are a reference, or even, I mean, yes, it is a reference, but it's much more nuanced than 
that. It's a kind of reference, it is supporting evidence, it is refuting evidence, it is that 
qualifying information, it is an instance of that. 

Stephanie Strickland: It's architectural. I refer you back to your architectural training. I 
mean, I might have a whole set of terms from architecture, or from music, in terms of which I 
could create a structure which is not based on citational reference. 

Yiliu Shen-Burke: Correct. And so, in that case, you might be talking about containment 
relations, where there is a taxonomy that you're trying to build up. And that is also absolutely 
possible in this work-in-progress that I’m building, or the future version of it. I haven't built it 
out yet. Ill show you in the next demo something that you know what it could look like. But 
just to, maybe, hint at where the possibility for doing that kind of relation-building comes 
from, I would just like to note that, this document itself can be seen as, for example, an 
instance of this Frode-type topic, in which I have placed a bunch of Frode-related blocks. So 
I could also create many of these containers in a nested fashion and start building up these 
taxonomies of meaning, and of information that's something you're talking about. I’m just 
going to go to the final demo, which will maybe hint at where this is all going. Boom. 
Stephanie Strickland: So, I think it would be nice if everybody could build their own kind 
of memory palace out of the material. Do you know what I mean by memory palace? 

Yiliu Shen-Burke: I do. And I would say that, even the paradigm I’ve just shown you would 
make that totally possible. So this is actually Prototype01, and I just want to show you, and 
the order is intentional here. So Prototype 01 is very different from what I’ve shown you 
before. This is actually an interface for navigating my personal Dropbox and here's there's 
one folder called SoftSpace Research. I’m going to select this Dropbox folder to be 
spatialized and you'll see what it's doing. So, this is why, I’m sorry, Stephanie, that I cut you 
off a little bit, I wanted to move on to this demo to show you what some of the possibilities 
start to be, right? So this is very simple. We're looking at folders and folders with images. 
And I’m just using that very simple directed-tree containment structure to build a three- 
dimensional force director graph, it's pulling in previews... 

Stephanie Strickland: Can you put your hand into there to twist that structure? 

Yiliu Shen-Burke: Yes. So I could, for example, say, "Oh, this branch needs to go over 


here." It's going to take a second because there's a lot of stuff to pull, but... 
Stephanie Strickland: Can you, then, put it all on the surface of a dome? 


Yiliu Shen-Burke: No, but by combining this and the previous demo I just showed, you 
would be able to, for example, indicate, "Okay. I want to see this topic, but laid out 
ordinally." Once I get to start working on it, I want to be able to write notes and place things 


in a specific order, that makes sense to read from beginning to end. And this is the point I 
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found earlier, while the rest of the content, which is not contained in that ordinal layout, can 
still float nearby and can adapt to the positioning of the things in the ordinal layout? So you 
can look around and say, "Okay. Actually, this other topic is also related, let me make sure I 
mention that, or pull something from there." Prototype01 doesn't have any text. I kind of 
worked backwards here because Prototype01 was very much about three-dimensional force- 
directed graph, hand tracking, pass-through video. Text was too scary for me, even at that 
point, and it took me a while to like, build up the courage, almost, to really tackle it, because 
what if it turned out that there was just no good way to work with text in VR? That's a major 
dead end to run into, but happily, I think, prototype two, three, four, proves that, actually, is 
very pleasant to write in VR. You have no distractions. You can place the text you're working 
on right in front of you. It's actually really nice. The only issue being, right now, the 
ergonomics of the headset need to be improved. I’m going to leave this running, just over 
here, and put my headset down with a nice close-up view of it. Questions? Prototype4 is a 
work-in-progress. There are a lot of things I want to like... I haven't finished building yet, but 
I’m very open to ideas, like the ones that Stephanie has been offering about how to connect 
all these technical and whiz-bang UI demos, back to real things that real people do to get real 
things done. 


Frode Hegland: Yeah, that's important. I have to have an initial question, actually. And that 
is, on the issue of collapsing, and especially in expanding. Especially in your previous view, 
there was a lot of stuff which is amazing. So what mechanism do you have for, "Okay. I’m 
dealing with this. I want that stuff out of the way right now."? 

Yiliu Shen-Burke: Yeah. None, on these prototypes. I was literally thinking about this earlier 
today. Because you do get to a point, pretty quickly, when you have a few hundred items. 
And by the way, text level of detailing, LOD, is another very interesting issue that arises in 
the 3D environment. Because you can't read text that's just far away enough that the letters 
are too small. But collapsing things, I don't fully know the answer yet, but probably it will be 
via the topic containers. So probably you will do your querying in a sense, or highlighting the 
content items by saying which topics I want to keep visible, and which topics I don't need to 
be distracted by at this point. And maybe, by turning on or off certain topic containers, the 
nodes themselves, any content item that is not directly, or maybe two degrees of separation, 
related to those topics, would, probably, not disappear altogether. You probably want some 
visual indication, "Oh, there's something over there." But it could be a small icon 
representation, instead of the full content itself. But I don't know the answer to that, because I 
think the best approach will make itself known through people actually trying to use this 
thing, and then realizing, "All right. I need to hide certain kinds of things away." And what is 
the logic, even, to decide which things to hide, and which things to keep in the workspace? 
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Frode Hegland: Okay. And then, a tiny question before I’m sure there's going to be billions 
from everyone. In your AR use here with a pass-through, the background on my screen, at 
least, is monochromatic, slightly warm. Is this something that I will have the ability to decide 
how it's rendered? 

Yiliu Shen-Burke: So the fact that it's monochromatic is the hardware limitation of this 
headset. Very happily you can see in all the promotional videos that MedVa has been 
releasing about their upcoming hardware, and also on the various leaks on YouTube that Meta 
and also, rumour has it, Apple, all these billion dollar projects are leaning very heavily into 
full colour, high resolution, low latency pass-through, so. 

Frode Hegland: I’m kind of leaning in the other direction because the normal pass through 
it's kind of harsh. So did you tilt it a little bit to reduce the contrast? 

Yiliu Shen-Burke: Yeah, I did. I dramatically reduced the luminance levels, and also reduced 
contrast a little bit. Because it is impossible to focus on free-floating digital content when you 
also see your full colour room around you. So when that pastel gets much better, I’m sure I 
will still have to tone it down to make it usable. 

Frode Hegland: Do you have an interaction feeling for how that should be done? Because I 
could imagine, with the next generation hardware, you'd want to be in really great resolution 
to see the room, and then, you want to tone it down. Have you thought about how 
(indistinct)? 

Yiliu Shen-Burke: Yeah, I have. So the short answer is, yes, because to make a lot of the 
promotional videos for this these prototypes, I’ve actually used a dummy 3D environment in 
Unity and so it's not this grainy video of my kitchen, it's actually this fake pleasingly 
rendered mountain cottage. So there it's, of course, by default, full colour. And I had to play 
with the settings a little bit to figure out what you have to do to the pass-through video to 
make it not distracting. And it seems that just by reducing the luminance, the exposure value 
quite a bit, maybe cutting it by half or by two-thirds, it's enough that your brain can 
distinguish background to the foreground. I mean, that's the main issue, that you need to 
quickly, automatically distinguish foreground virtual content from background irrelevant 
information. 

Frode Hegland: Yeah, that makes a lot of sense. And in 2D we have drop shadows, outlines, 
and all that kind of stuff. But what I’m wondering is, have you thought of the interaction? 
Does the user do this thing? Twist their hand? Move their head? wWhat might be natural? 
Because what we're seeing here is something that we're going to be doing 100 times a day in 
the near future, I think. 

Yiliu Shen-Burke: Are you asking about how to turn on the filtering for the pass-through? 
Or maybe I’m not fully understanding the question. 
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Frode Hegland: I’m saying, I’m in a room, looking as high-res and beautiful as I can, and 
I’m working on something which is really special, and I want to dim the light, so to speak. 
Have you thought of human-computer interaction for that? 

Yiliu Shen-Burke: No, I haven't. Because the paradigm, right now, is that, you would open 
up this particular app to be able to work with this content, and therefore, there's no provision 
in my app for running without the background filter on, because there always is going to be 
SoftSpace virtual content. But if I think, at an OS level, then there would need to be 
something. Yeah. I don't know. I actually haven't played with ways to control that reality 
modulation using user-controlled gestures or inputs. 

Frode Hegland: I have so many more questions. But I see Peter, and then Stephanie. But I 
think reality modulation, guys. That's a t-shirt. Peter? 

Peter Wasilko: Yes. Would it be possible to generate a fog effect? So it would be like the 
room but in a heavy ground fog? 

Yiliu Shen-Burke: I guess you could. So the upcoming hardware is supposed to have better 
depth detection, which is something you would need in order to map a Z value to the different 
parts of the path through video. So because with this headset, it doesn't know which parts of 
this image are further away, and therefore, should be foggier, and which parts are closer. But 
that's actually a great idea, because what you really want is, you want to be able to see, well, I 
mean, so you may notice that like this object, this virtual object, can actually float behind this 
table, for example. And that's, by design, intentional. I just think it's going to be too 
constrained to design a design interface that's really reliant on physical features in your room. 
What if you're in a hotel room and you don't have enough space to work on your PhD 
dissertation, which is massive? I think that would be really problematic. So part of toning 
down the background is also to make it easier for your brain to accept that this virtual object 
can be floating behind the desk. And it's okay. It actually looks strange at first. It doesn't 
cause any physical discomfort or anything, which it would if the background were full 
exposure. 

Frode Hegland: Stephanie? 

Stephanie Strickland: A new thing just occurred to me, it seems like you should be able to 
just select out that floating tree of things and have the background completely disappear. I 
mean, because I’m just looking at the selected thing, so that would tone it down. But my real 
question was, at what point can we expect hardware that isn't any more difficult to wear than 
a pair of glasses? Or if, almost, a pair of goggles? 

Yiliu Shen-Burke: | think a while. What I do know is that, from all these YouTube leaks that 
happened, things are coming quite soon, in the order of, definitely not years, so sooner than 


that from all the major players. New hardware is coming that has a much greater emphasis on 
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ergonomics, comfort, the balance of weight on your head, and pass-through that will actually 
make it feel less socially awkward to be wearing a headset and to have your eyes covered in a 
professional setting or in a social setting. It's going to be incremental. I don't have any insider 
knowledge on this, but from what I’ve seen in the press, see-through augmented reality 
display technology is incredibly difficult, to the point of maybe physically impossible. So 
that's going to take a long time and I don't know that that's necessarily the end game. It might 
just be that we end up with really high quality pass-through displays, because you can also do 
a lot more with pass-through displays because you have full control of the colour of every 
pixel. So you can have occlusion, you can have filtering, and you can have stuff that is really 


difficult in smart classes. 


Stephanie Strickland: But pass-through, that does not allow me to change my location, my 
visual location, at will, between this room I’m in right now, which you don't see, and the VR 
location. 

Yiliu Shen-Burke: Well, it could. Because you could have selective filtering. I haven't seen 
any great interface design examples of this, but you could have a virtual room that had 
windows through, which you had the pass-through coming through. So you could peek out 
the window to see what's going on in your physical kitchen that you're sitting in. 

Stephanie Strickland: No I don't want to peek out. I want to look out and see where three 
small children have disappeared around the whole space. I want to actually see the space, and 
then I want to see this space. I want a choice of spaces. That's what I want to be able to have. 
Yiliu Shen-Burke: I mean, my mind is going to all sorts of things. 3D scanners set up in 
your house that you could then see a recreated theme all over your house. But, I mean, at 
least a pass-through would be much closer to offering that possibility than to the see-through 
headset. So, yeah. 

Brandel Zachernuk: But, Stephanie. One of the things that I’ve played with in the past is 
having passed-through portals, where you place objects in your virtual reality environment 
that are stand-ins for where you always want to be able to see specific things, so you can have 
a doorway. I placed a specific persistent pass-through portal over my keyboard, such that I 
have the ability to be able to see that. So once you have an overarching capacity to alternate 
those things, there are all sorts of different ways of attenuating the virtual view, so that you 
have the ability to see the different pieces of it. 

Stephanie Strickland: That would be great, the attenuation. In other words, you could dial it 
down, right? If I just had an analogue dial that would remove the virtual view completely 
from my view, so that I now had my default, realistic view. But then I could just turn it up 
again so that I could, you know, now I’m back seeing the VR view. 


Yiliu Shen-Burke: I'll give you a functional example of this. If you step outside the safety 
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boundaries, the Oculus headset will just show you pass-through. So I’m trying to show that 
I’m just moving the headset outside the safety boundary, and you can see the virtual object 
coming in and out. 

Stephanie Strickland: You have to take that off your head to do that, right? 

Brandel Zachernuk: Yeah, it's simply a function of where you're standing. Because in the 
parlance of the system there's a safety boundary in which you've designated. You promise 
nothing's going to come in, and you're going to smash a vase or hit a kid in the face, unless 
they come in, in which case, you know, it's sad for them. But, yeah. One of the things I think 
it relates to, for me, is the idea of having an application in the sense of a space or a context 
being applied to something. Because we talk about apps, and we forget where the word 
comes from, but this is an app in the sense of applying oneself to a particular directory 
structure, and you can make all kinds of different inferences about what application you are 
considering at a given point. It could be that you're looking in a certain place. It could be that 
your hands are posed in a certain place. So there will be a number of ways of mediating and 
making determinations about what to do. 

Frode Hegland: The term used earlier, reality modulation, comes into this, because what 
Brandel was talking about is you can cut a hole to see the reality there, maybe where you 
have your coffee cup, if you're in a fully VR environment, which is quite useful, or your 
keyboard. But also, we can pipe in other things, a lot of us have video doorbells like the Ring 
doorbell, so if you wanted to, you could say, on this wall in your room, where you currently 
have a picture, a painting, remove the painting and have that doorbell, so you will always see 
the front of your house, for instance. So the notion of having a window, because a normal 
window, with VR, would normally be blown out, because there's too much light coming in. 
So why not replace it with something else? And then you get into, when you're talking about 
that kind of reality modulation, about different spaces, if you're at a home office, or if you're 
in an office or a coffee shop, you may still want the same information on walls. But if these 
are places you use frequently, you can design, you can tell the system, this wall is always for 
messages, this is always for timeline or whatever it might be. So when you go to the different 
rooms, it'll shift a bit, but it'll still be mapped onto. 

Stephanie Strickland: But it's always just as much as a window, right? It's not a 180 or 360 
view? 

Frode Hegland: Well, you can choose. VR normally is a completely synthetic environment, 
which is either 3D generated or can be based on photographs or whatever you want, so you 
can choose those. I think, also, the discussion here is, that sometimes you want to know 
what's in your physical environment, and one aspect of that is simply to see the video of it, 


another one is to have it rendered in 3D. You won't knock into the desk, because the desk is 


519 


indicated, but the desk may look completely different. Maybe it looks like you're in a 
nightclub, or in a jungle, or something, whatever you think might be fun, but you still have 
the geometry, so that you don't just sit down, you can move around your environment. So 
there are a lot of options for choices with that. But the thing that I just wanted to mention is, 
that you talked a little bit about Markdown and so on, which is interesting. So one thing 
we've discussed here in this group, which is actually part of my PhD thesis, well, it is my 
PhD thesis, is the notion of Visual-Meta, which is super simple. A PDF document normally 
doesn't have any metadata. It can, but normally it doesn't. A normal book, one of the first 
pages, it has a page of printers information, copyright, and so on. Are you familiar with the 
BibTeX standard for notation? Doesn't matter. So just imagine that you download a PDF, at 
the last page, there's an appendix that says, Visual-Meta, and then you have, author equals 
and then a name, date equals and then the date. It can also have the structure of the document, 
headings, or so-and-so, references, or so-and-so, all these things are there. One of the things 
that we've discussed, very much with Peter Wasilko’s input, is how that can be extended with 
further appendices. So a document that goes from your normal word processor with this 
information into your environment, you can then choose to put let's say, the glossary over 
here, the references. You do whatever you want. But when you go out, all those spatial 
representations, which is why it was so great to hear you talk about the coordinates, is then 
encoded. So you can keep working in a 2D document. Do whatever you want. But when you 


then go back in your environment again, all these things will snap back into that space. 


Yiliu Shen-Burke: Yeah. On a general level, coming up with some, sort of, legible mapping 
from something like metadata relations, or some other non-inherently, non-spatial qualities of 
the 2D media that we work with all the time, so in this case metadata about text or about 
PDFs, a legible and reliable mapping of that to a 3D representation, or spatial representation 
that you can manipulate, and you can make sense of and see very clearly in front of you, but 
then, when you take the headset off, you go back to the original file, those manipulations 
have resulted in the changes that you expect those manipulations to enact. That is, in general, 
I think, really going to be a very powerful quality of these spatial interfaces that you will be 
able to quickly, intuitively, kinaesthetically make sense of this metadata, that usually, on a 2D 
interface, is at best, just listed out and that works hidden away or doesn't exist at all, as you 
said. 

Frode Hegland: Yeah. It's something I’d love to continue the discussion with you at some 
other point. I see three hands. The first one I think is Fabien. 


Fabien Benetou: Yeah, I was wondering, we discussed about the transition of the boundary 
between the real or modularization of that. I was wondering, using the inverse of a window, 


so that you have either your normal display or the e-ink that you stick on the wall and that 


520 


shows the lens you add to your notes. For example, here you have the mirroring of your 
Quest, so it's a prompt to dive back and see that space. But I imagine you don't have it always 
on. But as you have that position in space, you could save it in this space. With the e-ink 
device, you can even physically print it if you don't think you've done it a lot from a window. 
And if you have, let's say, the virtual world, that's your point of view of the headset toward 
that virtual world. And maybe you have a mirror so that I can't put my e-ink here, that would 
not be very convenient in my office, but if I stick it there on the wall, then I can actually leave 
it. And it's again, always a problem to reconsider how you organize that space. So I’m 
wondering if that could be an interest in this area, you have your organized virtual space, 
because most of us have a desk with documents on it, and that permanence is pretty valuable. 
And we also have libraries or bookshelves behind us, and we do like to reach behind, and 
somehow, organize it, another reference we have. So I’m wondering, there is the beauty of 
having this infinite space you could reconfigure, but somehow, it's hidden away. If you put 
your Quest on the side, then it's all hidden. You, of course, have your desktop, you can just 
start with a prototype of just changing the desktop, or having a window there. But I’m 
wondering how being disconnected from the whole desktop would be interesting, having this 
virtual permanent window of that organized base? How we would feel? I have no idea, but 
that should be interesting to try. 

Yiliu Shen-Burke: That's a great term, this a prompt, this little reminder, like a magic 
window back into this garden. The garden of ideas that you occasionally glance at. I think not 
now, but it's reminding you, "Okay, it is there." I mean, the practical question of how to 
access this information when you don't have the headset on you, or if you want to share it 
with somebody who doesn't have the hardware, I think, that was, very much, the underlying 
motivation for the experiment I did with the ordinal spatial layout. Because if you have such 
a layout, what you could do is, for example, always be reading and writing back to a Dropbox 
folder, and to Markdown files in the Dropbox folder. And many of these note-taking tools 
like (indistinct) and Obsidian actually just work with Markdown files in exposed folders in 
their local first applications. And so, if that's the case, then your reminder to go back to the 
headset could be that you are actually working on the exact same notes, most of the time on 
your laptop, but when you need to look at a bunch of images... Visual content is really 
difficult to make sense of in these interfaces. Or when you have just finished doing a brain 
dump, and you need to see, "How can I, in five minutes, just make sense of all this?" And if 
you put the headset on, and there is this reversible mapping from the 2D information, the 
ordinal information, and it's metadata two or three representation, then you can just put the 
headset on, and be looking at exactly your Obsidian notebook, do some stuff, move some 


things around, take your headset off, and then, when you in the headset, create a little outline 
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of like, here are the three categories, here the three relations between them. And when you 
take your headset off, go back to your laptop, it's there. But I think that's a more practical, 
sort of, you need to be able to do this, for such a tool, to be at all practically usable. But that's 
separate from, Fabien, your suggestion. And this very lovely, poetic suggestion of having a 
magic mirror view that's always reminding and subtly updates itself in response to changes 
that have been happening in the 3D workspace. And it even might just be that, even being 
able to quickly glance at the global structure, for example, even this thing that we're looking 
at on the screen right now. I don't really know what it means at the moment, because I haven't 
really worked in it, as a 3D object for long enough. But if you have worked in it for an hour 
or two, you might start getting a sense of this cluster of ideas over here, this part of my 
research means there's a big distance between that and the next cluster. Maybe having 
something like that, just easily accessible, would be enough to trigger some of these 
memories that you will have developed working in a 3D space. 

Frode Hegland: So you, kind of, almost, made a throwaway remark that this is a little 
practical thing, or whatever. I think this is probably the most important thing we can discuss. 
Because we'll look at how best to use it, that will be iteration, testing, and discussing, but 
how to be able to share it. It is so important to take it in and out. I have a word processor 
called Author, and Brandel has been kind enough to dig into my files and to provide access to 
a VR view. So that's a nice little hack that, if we can provide an ecosystem where these things 
can be used, it is the dream. It would be absolutely fantastically wonderful. I see Keith has 
joined us, which is lovely. I’m not going to tell anyone where he works... Meta... Anyway, 
Peter? 

Peter Wasilko: Okay. I have a few links on the sidebar. The first link that I dropped in was to 
an NHK program. It was a cultural heritage piece that had a segment in it on 3D imaging of 
temples and historical cultural sites to generate VR replicas of them. It's a very nice video to 
watch. And about, maybe, halfway to two-thirds-ish in for that segment of it. Then, I dropped 
a link to Mark Bernstein’s essay discussing typed Hypertext links, which also seemed 
relevant to today's discussion. Then, I included a link to a small excerpt from the 1990s 
movie Johnny Mnemonic, depicting the use of a VR headset for accessing the internet. And 
the interesting is, the time scenario was targeted as it being 2021. So it seemed incredibly 
timely, that back in 1995, they anticipated that 2021 would be the exact time that that 
technology would be coming online. So you might enjoy watching that short little video clip. 
Then I put in two links related to BibTeX, and I finished out with a link to Michael Benedikt's 
paper re-examining some ideas from their seminal book, Cyberspace First Steps, which was 
an MIT Press book, currently out of print, that I’ve talked about many times in here, which 


introduced the idea of the unfolding of spaces in VR. So you can be looking at three spatial 
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dimensions that will be associated with three parameters of a higher dimensional object, and 
you then, be able to select a point within the first three, that would effectively, either 
correspond to an actual object or to a data point, allowing you to open it to a whole range of 
other objects that correspond to, basically, having those first three search parameters at the 
value of the location you set. So, in this case, instead of positioning objects in VR, your 
positioning is the query to access the higher dimensional object, and then, unfolding it, which 
was depicted, almost as if, a second cube being opened with half the face cut away so that 
you're, basically, looking in on an open cut-out slice of a box, in a flipped orientation 
depicting three additional dimensions. And you'll then be able to do that. Another idea that 
I’ve been kicking around for a long time in Hypertext systems, would be to have a mirror 
mode, where you'd be able to reverse the nesting structure that you'd use to traverse to a 
given point. So you'd be able to effectively turn around and look back up through the 
containment hierarchy that allowed you to reach the node that you were currently at. And 
being able to toggle on and off a mirror node, strikes me as a very useful affordance that I 


have yet to see in any systems I’ve had to work with. 


Yiliu Shen-Burke: I’m going to spend the next week digging into all of these. One point that 
you mentioned about mapping the actual position, the cartesian position of a node in some 
values, or some metadata, that's very close to what I’ve been sketching out for prototype 
number five, or I don't know which number it will be in the end. But a way to represent 
properties, for example, and there's metadata of these content blocks. For example, if it's a 
quantitative property, then it's very clear how you might map that to a spatial dimension. But 
even something like a Boolean, has this image already been edited or has it not, right? You 
could do all sorts of things in terms of mapping it to positions. You could snap it to one plane 
or another, depending on the Boolean value. You could also represent the Boolean values 
true/false as two free-floating nodes, in and of themselves, that try to pull the things that are 
true closer to themselves, and they try to pull things that are false closer to themselves. And 
then if you had five, ten, probably at a certain point it would stop making sense, but if you 
did multiple properties like these, so you did a couple of Boolean properties, you did a couple 
of like numerical properties, and you just let them all settle into the configuration that the 
whole system wants to. I mean, it's like a principal component analysis but really rough, and 
maybe, interactive, manipulable, and really difficult to understand, because it's not, at all, 
mathematically rigorous. But for some collections of information, maybe you're writing a 
paper and you start adding some quantitative properties, or toggle properties to things, and 
you just want to see, "Okay, I’m kind of stuck here. I just want the system to tell me what 
shape can these bits of information take if I prioritize these properties or de-prioritize those 
other ones." That's definitely something I’m thinking about for prototype number five. But 
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that's another parallel track altogether. I just wanted to kick it back to everybody a little bit, 
because Frode sent an email a couple of weeks ago talking about the question of how to relate 
text and information more generally in a 3D environment, one to another. Either the question 
of like literally the visual representation of the relations between things or a more conceptual 
level. So something that I’m struggling with, but definitely I’m in the thick of it, at the 
moment, with prototype zero four is how to represent, and how to think about the relations 
between blocks and text to each other, blocks of text to topics, blocks of text to multimedia, 
like images or URLs. Stephanie made a very good point that either all relations are references 
because it's a reference, but then, you should specify what kind of reference it is. Or 
references but one of many possible kinds of relations. I have a lot of conversations with 
people about semantic triples, in relation to semantic web, recently, but specifically to text, 
which is very slippery as well. It's natural language. There's interpretation involved. How 
have you all been thinking about, or what examples have you seen of ways to relate things? 
I’m thinking all the way back from Ted Nelson's, Xanadu style transclusion indicators, to, I 
don't know, something like this prototype I’m showing here, with these radial links and lines. 
Frode Hegland: Fabien? 

Fabien Benetou: | want to step back just a bit on, it's going to sound a bit harsh, I find your 
work very interesting, but I’m probably never going to use it because I need to make it 
myself. I think the research you do is interesting, but I don't think you can explore all of it. 
And some, for example, of my quirky ideas, I think nobody is going to explore. But why I 
find and rightfully so, but I still going to explore it. And then, that's also why I like 
programming, is because I can have some really strange ideas, and maybe, nobody should 
explore them, because it's going to be useless for them, but I want to be able to do that. So the 
problem I have is, if I use somebody else's system, at some point, I get stuck. I don't want to 
reinvent the whole wheel, and I, obviously, can't even do that. So what I’m going into here is 
some of the idea both of us have, some of the ideas are not shared. One of the beauty of the 
power of tools like Notion or all the PKM-PIM trend of the moment, is also that some of the 
effort is being distributed through the community. So I’m just wondering, are the 
components, patterns, or things for example that make Unity so famous or popular that we 
should do together? That we, as a community, should have, maybe, I don't know, 
implementing some way to explore Dropbox or Google Drive or whatnot, we don't have to 
keep on re-implementing that? Or maybe, some way to spatialize? We don't have to so. I’m 
just thinking it completely naively. I don't have an answer to this, but I’m wondering, what 
are the things we should re-implement from scratch because we need to dig there and there is 
no answer? What are the things we should not reimplement? Do you have some patterns? Do 


you have some plugins, or a cookbook, or recipes that you want to rely on, that also, maybe, 
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you don't want to explore because you don't find interesting, but you know it could be 
interesting and you want to rely on this? So I’m wondering, at the larger scale, a community 
of people who are interested in managing knowledge, writing, or reading text here, how can 
we be a bit more strategical about the work we can do? 

Yiliu Shen-Burke: That's an excellent question. And I think answering this question is going 
to make all the difference in whether things like what I’m building, can become useful. 
Nothing good enough that there is mainstream adoption or not. I mean, I think the shorthand 
answer is open sourcing some or all parts of these projects. I’m in conversation with Brian 
Eppert of Noda, which is a VR mind mapping application. He's brought up a lot of similar 
points, which is like, we are all either reinventing the wheel, or we're relying on very poorly 
adapted frameworks that come from the gaming world, where, essentially, you're given 
options to fire weapons at your PBS. And there probably is a lot of, well, I would say there's 
going to be increasing duplicated effort. Now that it seems that people are actually tackling 
this class of applications seriously, I would say, even a couple of years ago, there wasn't 
necessarily much duplication because everyone was only making games in VR. So often I 
had things like, I had to re-implement, by hand, an image processing library, and I had to 
implement, by hand, an image texturing LOD system. TextMeshPro is already pretty good, 
but I had to definitely tweak that a lot to make it usable. And so there are a lot of these things 
that if you're like developing a web app, you would never, ever, in a million years, in 2022, 
want to build by yourself because there you'll have a thousand excellent libraries already 
available. And that's just not where we are. But I think, not duplicating effort while, as you're 
saying, being mindful of the areas where it's productive to just dig on your own because 
there's much more there than has already been discovered, balancing between those two. But 
I think, I’ve just moved so much more slowly than I would have liked, with building the first 
version of the app, version 2020, the one with the big white room, because of the need to just 
reinvent a lot of these building blocks that exist in abundance in other platforms. I have many 
of them now in the toolbox, and that's been great, and that's one of the factors. And these 
prototypes I’ve been showing you, I started building in February, and so I’m quite happy. I 
could be faster, but I’m pretty happy with how quickly they've come together. In large part 
because I had done years and years of work. The pinch, the spatial cursor thing took, I don't 
know how many months to really figure out and built, and that's the kind of thing that I also 
would be happy to promote as an alternative paradigm to the larger community. I mean, there 
are issues with it, we have to tweak and all this stuff. But just as an alternative to the laser 
pointer, which just has so many issues, both ergonomic and conceptual. To me it's a mock 
weapon, right? Those are examples of things I’d be more than happy to have other people 


adopt, offer feedback on, and if it's open source, to other people to improve on at a much 
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faster rate than one party would be able to do. So Fabien, basically, yes, let's talk about that 


because I absolutely don't want to reinvent the entire operating system from scratch. 


Frode Hegland: I see we have two good hands coming up here. But just really briefly. I 
share your frustration about doing something commercial, and also being part of a 
community. It's really hard to decide what's going to be your secret sauce, so to speak. And 
what has to be shared. My initial feeling with what you're doing here is, data in and out 
should be shared and working on protocols, that people should be able to choose your 
environment and you have your amazing interactions. It's a stressful and longer discussion. 
Brandel, you're first. And then, Bob, you're second in the queue from now, yeah. 

Brandel Zachernuk: Awesome. So in an answer to your question of how you apply spaces 
and attributes to data, I dropped a link in the chat to a researcher, who used to be at the 
University of Monash, now at the University of Queensland, Maxime Cordeil. He did these 
immersive analytics, actually, I’ve shown in this group before, but it's really phenomenal, in 
terms of being able to show multivariate data. And more generally, one of the things that I 
think is really valuable, is having the ability to really, rapidly, and substantially alter the 
arrangement layout and visual appearance with gestural manipulation that has a quick and 
one-to-one impact in terms of scaling things, or moving things, or colouring things, changing 
aspects of their objects velocity. Those things that are the most detectable to us, in a lizard 
brain kind of way, but once that we can, actually, modify and sculpt the presentation. Which 
is, somewhat, at odds with another really important thing for us, which is the persistent 
speciality aspect. but there are interesting ways of trying to square that circle for individuals 
doing different things. The question that I had for you was, with all of these prototypes, 
including back to 2020, but also your one through four, have you sat with them, used them, 
and thought about the impact that they have on what you do? Like with this beautiful tree 
view, and I’d love for you to move your headset around, and talk a little bit, especially for 
Bob, because he became about what this is, I think you'll get a kick out of it. Has it changed 
what you think of, or how you understand what these things are, and what you want to do 
with them, as a consequence? 

Yiliu Shen-Burke: That's a great question. So to eat my own dog food, on a practical level, 
can I use these tools I’m building to, actually, do some very minimal amount of the work that 
I proposing people to use them for. And so the blog posts that I published for prototypes two 
and three I wrote, the first drafts are in the prototypes themselves. And prototype two, I wish 
I didn't show you because it was a really dumbed-down version of prototype three, totally 
focused on text editing and writing. But my benchmark for success for that one was if I could 
sit for one hour and just write. And at first, I thought I could qualify this with like, "Oh, can I 


write well? Or can I write X number of words?" It actually turns out that, if you can't write 
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comfortably and productively, you're going to stop after 30 seconds, because it's just the most 
frustrating thing in the world to try to write something with interfaces getting in your way or 
whatever. And once I got to the point where I could actually write, and it was like, an 800- 
word draft I think, and published, it was 1200 with additional comments and images. Once I 
got to one hour mark I knew, "Okay, that's pretty good for me. I’ve surprised myself, that I’m 
convinced you can write comfortably in your VR headset." Which, again, I was very scared 
of the answer to that question when I embarked on that process. And the higher level question 
of, have I changed how I think about what this is, what this information is? Absolutely. And 
this is all just making me want to go back to work and finish building prototype four, 
because, are any of you familiar with Alexander Obenauer's Itemized OS? I will post that in 
the chat. So the Itemized OS is a proposed paradigmatic design for a computer operating 
system, where the lowest level primitive is an atomic item which can be any, sort of, block of 
content. So a block of text, an image, a PDF. It can also be a composite of those things. So 
you can have a calendar event that's a text at a time, et cetera. And these atomic units are 
infinitely, flexibly recombinable into different configurations, and they can be contained, and 
the containers can have certain logic about what they do to the containees, but 

the containees are only ever temporarily contained within the container. It's not that if you 
delete the container all the items go away, as well. So, I guess, one of the differences between 
the Itemized OS and the OSs that we actually use, that are actually out in the world, is that 
items have a primacy that files and folders don't. And files and folders have a definite 
location on your computer. The containment structure is a tree. And the Itemized OS it's a full 
graph. You can have any sort of (indistinct) pointing any which ways. Which, to me, does 
sound a little bit overwhelming, potentially for someone who just wants to check their email, 
and make sure they get to their (indistinct) on time with their calendar app. But maybe as an 
underlying layer that would give power users or developers all this flexibility that they 
currently don't have, it's not a bad idea. And all the examples that Alexander Obenauer gives, 
and especially the visual mock-ups he's created of how this would work in the 2D 
environment always run into this problem of views, so give you a very concrete example, I 
think one of his examples is, you receive an email with a calendar invitation in it, and in the 
Itemized OS, the calendar invitation would be its own item that you can move into your 
calendar, and the calendar invitation has this existence independent of the email that arrived 
in, and the calendar view that you're seeing in relative to other events. And in the mock-ups, 
you see the email as a window with the event in it. And then, you see the calendar window 
with the event in it. And you're always jumping between these views, even if the data is 
living in the same spot, in IPFs or whatever this is. The way the interface represents this is as 
different things that are in different locations, because you know from the physical world, 


we've learned that a cup that is in my apartment right now, cannot be the same thing as a cup 
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that is in Brandel's apartment, because, almost by definition, that makes them different cups. 
Whereas, in the Itemized OS, they can, and should be the same item. And so, with 
Prototype04, right now, the missing step is that that document container is actually just 
another type of item that has been expanded, and has pulled in all the other items that are 
contained in it. You could collapse it, and then, everything goes back to the force director 
graph structure. Or open up another container that would, then, temporarily take over the 
layout. And all the items that it contained would, for brush into it, and so you have this literal 
continuity of existence of items, which have a canonical representation of each canonical 
thing. And the reason why I’m calling all these things projects is because I have no idea if 
this is a good idea or not, actually, in practice. But I’m pretty sure, in some use cases, it is an 
excellent idea. And a concrete one is coming from architectural design, I have a lot of early 
users who are architects themselves, because I harass them into using SoftSpace. And when 
you're doing visual research, you're doing report, you don't want to, actually, duplicate... 
Images are interesting because they are very immutable, they rarely change. So you really 
have one image, and you don't want to make a copy, because you're not going to make 
changes to that image if you move it into a folder or if you want to reference it in a different 
part of your project. It's the same image. It's the same core idea. And so for mood boarding, 
as a use case, or for any visual heavy research, this ability to, actually, see the same idea 
represented by a canonical visual representation of it, moving between the different places 
where it is playing a role in your project, I hope, I think could be very powerful and actually 
useful. So this is something that has only occurred to me, Brandel, as a response to your 
question, after having played with my prototype three, and then asking... I had a call with 
Conor White-Sullivan at Roam, and we butted heads over references, containment, and 
things, and afterward, I realized, "No, Conor is right." I just couldn't see the possibilities that 
my prototype was presenting, just because I was so stuck in the items in folders living in only 
one place, and therefore, to show that thing somewhere else it had to be a sim link, or it had 
to be some other type of relation. But, actually, no, it could just be the same kind of 
relationship of being contained in something, or not contained, but referenced in something. 
So, yeah. That's one small example. I hope to find more. But also, the reason why I’m really 
excited about this one, in particular, is because I’m wary of these insights that might come to 
me, that only occur to someone who has worn a headset and tried these things. But the fact 
that I’ve come to this idea for certain information architecture, and then I find examples of it, 
not only in contemporary writings, like Alexander Obenauer's, but also going all the way 
back to transclusion. What's transclusion? This is transclusion, right? That makes me really 
excited, though, it feels like, okay, this is two ends of the donut finally coming together. We 


had the conceptual possibility and the technical possibility. And one was really far away from 
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the other, but now they're finally able to touch. 

Frode Hegland: Bob, finally your turn. 

Bob Horn: Thanks, Frode. Sorry I was late to this session. I had eye surgery yesterday, so I 
had a post-op this morning. I’m interested in several topics back now, and if I’m mistaken, 
please, just pass on to what you're interested in. But I thought I heard that Yiliu was interested 
in the relationships between different parts of text that might have to do with content that 
goes beyond metadata? And if that is a topic, then, I would be interested in having a 
conversation about that. 

Yiliu Shen-Burke: Absolutely. Just to frame it a little bit, and Bob, I want to hear what you 
have to say on this topic. But the framing is this, which is, if you go looking for information 
on the semantic web on the internet, as I did, you come across this core concept of the 
semantic triple, which is, an object, a predicate, and a subject. So Eve eats the apple, or 
whatever. And to me, and the semantic web painted this picture of a web that was, basically, 
full of these semantic triples, and you could do all this processing, querying, and automatic 
deduction based on these semantic triples. But what always struck me is this like, people who 
work on semantic web themselves acknowledge, is that, it's obviously very labour intensive, 
every single relation between one thing and another has to be clearly, manually defined. And 
then, all these questions are made up of aliasing, what if the verb and the noun version of that 
concept should really be the same thing? Or just gets really messy, really quickly. And also it 
was unclear to me, in the system, what level of abstraction this meaning should really live? Is 
it something that's very low-level, technical, pervasive, machine legible? Or on the other end 
of the spectrum, is it something that is almost, by definition, human and poetic and it will 
vary subtly from one speaker of the language to another and be completely open to 
interpretation? Which is why I think for example, Roam, and its, sort of, clones have settled 
on this typeless reference, where you just point a block of text to some other concept, and 
there's just this directed reference, and there isn't a way to get, I don't know if it's in the 
works, to define what kind of reference that might be. Whether it is supporting evidence, 
refuting evidence, or an instance of something, et cetera. So then it got me thinking, "Okay, 
well." One is, if that were the case, if you did have typed references, does having a spatial 
interface open a new possibility for representing them? For example, you have a line from the 
object block to the subject block in this line with the predicate displayed in the middle. That's 
a very simple example. Is that even useful or would that be overwhelming? And then I 
thought, "Well, maybe we're thinking about it backwards. Maybe the block of text, itself, is 
the predicate." Normally you would be writing a sentence like, "Yiliu works in VR." This is a 
sentence in the text block, and Yiliu might be one topic, and VR might be another topic. And 
it might actually be that Yiltu is the object, VR is the subject, and this messy, squishy natural 
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language content object, which is not typed at all, it is what it is, it's literally that unique 
string of characters, is the relation, is the predicate from one to the other, and therefore, from 
one to the other it could be every place where those two concepts are ever mentioned in the 
same paragraph. That paragraph is a predicate. And then, you could take another level and 
say, any place where there's an indirect relation, it's also a deducible predicate between Yilitu 
works in VR, VR is an emerging technology, in which case Yiliu has this relation to emerging 
technologies. And then, once I got thinking of that, then I just thought, "Well, I’m not going 
to try to make sense of this. What if it were all just a force-directed graph and you could 
visually see it as being connected from the Yiliu node, to the VR node, to the emerging 
technologies node?" And I don't know what exactly that would mean, but it would mean 
something. And you would be able to concretely look at it, and do something with it. But I 
don't know. And that's just the train of thought that I’ve gone through. And I’m sure there's 
been a lot more work and research in this area that I’m not aware of. So I’m very happy to get 
pointers, references, and ideas. 

Bob Horn: What I’ve heard you saying is that there are maybe structures that have already 
been discovered that relate chunks of text that, maybe, one sentence to seven to nine 
sentences. Which is what I saw in your demo. The answer is, yes, there are such things. They 
were invented 50 years ago or more by me. There are, for example, in stable subject matters, 
the kind that exists in textbooks, procedures policies, and documentation, training materials 
in business, there are 40 such structures with a few loose ones at the end, some others. But 40 
stable ones, they've been used in business industry and government for the last 50 years. 
When I left being CEO and Chairman of that company that sold them, we had trained 
400.000 technical writers in business, in 30 countries, around the world. They all paid for this 
information. So that's one structure. I sold the company long ago, 30 years ago, and I 
presume it still exists. So, anyway, there are other structures that relate text like that. There 
are about 15 to 20 that I could tell you about, and outline. And there are a bunch more that 
need a lot of intellectual work to improve human thinking. 

Frode Hegland: So on that note, we have come up to the two-hour mark. And we try to keep 
to that. So this will be, now, uploaded, transcribed, and distributed. Any final comments? And 
to make it clear, you're very welcome to come back, and continue the conversation on any 
Monday or Friday. It's a very worthwhile topic. It sure is related to what we're dealing with, 
and obviously, that goes for all of you, not just the monthly meeting. 

Yiliu Shen-Burke: I have only a question which is like, can I come back to this forum? 
Because this is incredible. Fabien, thank you for the connection, and thanks to everybody for 
taking the time. And I definitely have the sensor you know a thousand other threads that we 


could have chased down and if I can come back on a more regular basis. Maybe have more 
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opportunities to chase them. 


Frode Hegland: So, let's have a thousand more meetings. Have a good weekend everyone, 


it's good to see you. 


pe 


€ 


Screen 1. Shen-Burke, 2022. 
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Screen 2. Shen-Burke, 2022. 


Screen 3. Shen-Burke, 2022. 


532 


533 


Yohanna Joseph Waliya 


Post Digital Text (PDT) in Virtual Reality (VR) 


With the aid of VR lenses, VR headsets as well as VR joypads, text in virtual reality (VR) 
blends itself immersively with its readers because both of them are datafied to be virtual 
embodiments, noticeably, in the Metaverse. Thus, virtual reality technologies (henceforth 
VRTechs) turn such text into experiences that make readers to interactively feel a sort of 
bodily astral projection couple with emotional trajectory into oneiric world. In other words, 
VRTechs algorithmically convert coincided literary utopias of an author into vivid 
experiences through Artificial Intelligence (AI), brainpower signals and the non-invasive 
Brain-Computer Interfaces (BCIs). Bansal and Mahajan rightly confirm that Facebook, which 
is now, Meta, has already advertised in April 2017 the development of its mind-controlled 
non-invasive BCI for typing using brain signals in order to make typing fivefold faster than 
usual by applying a functional magnetic resonance imaging (fMRI) signals to scan the brain 
many times within some moments so as to translate thoughts to text (5) on digital platforms 
just like the generative Artificial Intelligence that translates text to gif (T2G), text to 3D 
(T2D) simulation, text to motion (T2Mo), text to image (T2]), text to code (T2C), text to 
video (T2V), text to NFT (T2N) and text to music (T2M), we shall have thought to text 
(Th2T) in the VR. 

VR textual experience is equally an ekphrastic form of technological induced 
phantasmagorical, hypnotic and hallucinatory reading that capture readers’ soul, spirit and 
body then glue them to particular physical location at the same time whilst they are immersed 
in reading the text. In fact, the future 3D texts in VR will print automatically from the authors 
or readers’ brain and mind into the metaversal ecosystem in a matter of moments while they 
engage themselves in imaginary creative thoughts influenced by the text read or its prompts. 
Therefore, text in VR could be reified like gravel fetched by bricklayer to mould blocks. 
Thus, it will be seen, fetched and felt. It is obvious that digital poetics must be redefined to fit 


metaversal VR text environment. 


The digital poetics of metaversal literary text requires necessarily the knowledge of 
blockchain art or crypto art, three.js or babylon.js (JavaScript libraries), Web Graphic Library 
(WebGL), WebVR including Virtual Reality Modelling Language (VRML) to produce the 
procedural creative works. Nevertheless, the disadvantage of such procedural creative works 
in the Metaverse could be interactively hacked by hackers in order to manipulate readers’ 


thought pattern to suite some philosophical and religious belief contrary to their original 
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belief systems. This art is called “neuropsychological hacking”. In lay man language, “ brain 
and mind hacking”. Consequently, there is a tendency, in the future, authors and content 
developers of immersive storytelling may manipulate readers' propensity directly by editing 
3D VR text while integrating with it via the optical head mounted display (OHMD). 
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Stephen Fry 


In closing: A Prediction 


Language has the astonishing capacity to send and receive pictures, ideas and full dramatic 
scenarios to and from the minds of those who use it. Artificial Intelligence seems now to be 
developing the ability to do something similar - convert language into images. Dall-E and 
Midjourney are, at the time of writing, popular free examples of this. How text will be 
integrated in, exploited or harnessed by AR/VR and the like is an open question, but this new, 
or at least newly celebrated, capability of AI must, one presumes, be a part of it. 

But we only have to look back to realise how unknowable a future it is. Midjourney’s very 
name should remind us that technology is always moving. 

When I first showed friends a smartphone in the 90s (a Nokia Communicator or Sony 
Ericsson, probably) they thought them slow, cumbersome and consequently without apparent 


use. 


Twitter would suffer routinely in its first five to ten years from server outages, and cause 
DDoS type crashes to other sites if you sent too many people there. 

The apparent flakiness of the technology caused people to miss the real point of what such 
technologies could actually do to our species socially, psychologically, culturally - 
existentially. They were so hung up on the primitive early rollouts that they couldn’t see what 
the implications were. 

I say “they” I mean “we” of course, because I was as blind as everyone else. 


Then again, who looked at early Karl Benz automobiles and foresaw Formula One racing, 
four lane highways, multi-storey carparks, EVs, adaptive cruise control, drive through burger 
joints, decades of brain damage in children from gasoline lead-poisoning and rising pollution 
and climate change? They just saw noisy machines that were only for the rich, which required 
either a skilled chauffeur/mechanic, or a fair deal of knowledge concerning chokes, 
carburettors, jets, magnetos, spark plugs, double-declutching and the lord knows what else, 
just to drive a few miles. 

Who looked at Twitter, Facebook etc and foresaw the maelstrom of convergent storm systems 
that has swept like a destructive tornado through so much of human intercourse and comity 


over recent years? 
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It’s hard to trust any predictions from anyone. But just as the “paperless office” is a sick joke 
to anyone working in real offices, so is the idea of “keyboardless computer interfacing” - both 
always just around the corner, but always defied by our human liking (when it comes to 
virtual communication at least) for manipulating the visual symbols of language rather than 
deploying vocal utterance, text rather than speech. Yes, we have FaceTime and Zoom, but 
most people hate them and only use them to please their bosses or their mothers. We'd rather 
write a letter (as we should call emails now, surely?), or send a WhatsApp/iMessage. I 
suspect this will be just as true for those who like the idea of stepping into Zuckerberg’s 
Metaverse or wander about wearing digital spectacles of some kind of other... 

My only prediction is that everyone’s predictions will be wrong. Including mine of course. 


With this pleasing paradox, I will leave you. 


Mn 
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Graffiti Wall 


In response to the Editor’s question: “Do you have any thoughts on text and knowledge 
work in XR which you would want to put in the book?” 


Tom Standage 


Not really. I have not got Workrooms to work. My main thought is simply this: there has got 
to be a better model than Miro and Zoom. So I think there is scope for a more immersive 
approach. But that does not mean today’s vendors and today’s solutions are the right ones. 
Talk of the “information superhighway” in 1993 was directionally correct but none of the 


vendors that delivered the vision (Google, Netflix, Amazon) existed at the time. 
Martin Tiefenthaler 


Since there is no progress in humanism without reading involved, the main question will be if 
(in alphabetical order) ar/mr/vr/xr will technically and typographically be able to provide 
texts that are long enough to convey content that is telling enough, and deep enough, and 


encompassing enough. 
Ken Perlin 


For creating text, it's not clear to me that we will want to use a keyboard, either real or 
virtual, in a future where millions of people wander around together in a shared extended 
reality. Perhaps we will simply move away from the use of text altogether. 

After all, speech-to-text is now quite reliable, and in many cases is faster than typing. Still, 
there is something appealing about using our hands rather than our mouths to create text. It 
allows us to work with text while continuing our conversation with other humans, which is 
very useful for collaboration. 

Because of the recent emergence of XR at the consumer level, a lot of people are now 
thinking about the text input question. But what properties should a “virtual XR keyboard” 
have? 


One of the great things about using your hands to type on a QWERTY keyboard is that you 
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don’t need to look at your hands. You can keep talking with other people, maintain eye 
contact, be able to absorb their body language, all while typing away. 

I suspect that we will continue to value those two constraints: 1) the ability to continue 
talking with people while creating text, and 2) not needing to look at your hands while you 
are creating text. 


Exactly what form that will take, as XR continues to go mainstream, only time will tell. 


Bernard Vatant 


Got it, but never tried that kind of technology, and not eager to try. I've never supported 
headsets even to listen to music, too close to my ears. I rarely listen to music at all, actually, 
although it's a unique experience and I love it, but I need a lot of silence before, and after, and 
a lot of space around. And all those things are rare and difficult to find in this noisy world. 
The computer screen and the keyboard have been my ultimate concession to technology, 
because they still look like a page. But I try to go back whenever I can to paper, with my old 
fountain pen and bottle of ink. For me, text has the smell of violet ink, a childhood's smell. I 
have no smartphone, touchscreens (I had to search right now the English word for "écran 
tactile" which I had forgotten) drive me crazy, applications drive me angry. 

Augmented reality, or virtual reality, are arrogant and scary terms. There is so little we know 
about the real world, so much to discover in every corner of the real world, I could use the 
rest of my life to read every stone, every leaf of grass, every chunk of wood in my small 
garden, the way to move of every living thing I'm related to. 

I'm aware all this looks like the rant of an aging man, more and more a stranger in his epoch. 
This will not improve now, I'll turn 70 next year... some say this is still young age... but I 
already felt a stranger in my epoch when I was young... 


What else. Bon voyage vers le futur :-) 
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Stephanie Strickland 
Is it disabled, or unwieldy? 
Anne-Laure Le Cunff 


While I absolutely believe XR will impact the way we view, read, and interact with text, I 
don’t know for sure what that will look like. Traditional text has a ‘sense of place’ that 
doesn’t seem to perfectly match the one a user experiences in VR. People complain about 
how uncomfortable it currently is to read long texts in VR, and I think it has to do with that 
sense of place. 

How do you locate yourself in both a 2D text and a 3D world? Does it even make sense to 
force some artificial one-to-one mapping of those two mediums, or should we completely 


reinvent what text looks and feels like in VR? Time—and space—will tell. 
Stephan Kreutzer 


There’s apparently the natural tendency of obsessing about layout and presentation repeated 
all over again, while little is done in the area of augmentation, handling structure and building 
common infrastructure for knowledge work. A main benefit of text as a medium is that it can 
avoid or reduce the unnecessary distractions introduced by mis-applied visuals and in this 
way help with focus on the actual content. Unsurprisingly, the VR hype cycles don’t seem to 
contribute much in regards of improving how we go about our ever-increasing amounts of 


information. 
Phil Gooch 


Here is what I would like to see. I love the tactile experience of opening a book or a 
magazine. The physical medium. Turning the pages. I love the tactile experience of writing 
on paper, and also typing on a keyboard. 

If there was a way to combine that tactile experience - which is something almost universal, 
that we can all share, irrespective of any auditory or visual impairments - with some kind of 


augmented reality, then this could be part of the future of text. 
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But this would be beyond a 3D interactive visual hologram. We need to think beyond that 
towards something like the NeuraLink, where we have augmented thought and an augmented 
'‘mind’s eye’. 

We interact with a physical medium by touch. And we close our eyes. And a beautiful, 


interactive world opens up. 


Of course, this is science fiction now. But so was Douglas Adams' BabelFish forty years ago, 


and now a reality that we take for granted. 


David Lebow 


XR war rooms - virtual wall-size arrays and other technologies for multi-source knowledge- 


building activities. 


Jim Strahorn 


Text, Writing, Reading, Word Processing, Dictatinging or Talking Verbally ... on stone, 
papyrus, paper, screen, or in video, Virtual Reality or holography ... who knows ... not all of 


the above, but many ... in an uncertain world???? 


Esther Wojcicki 


VR spices up the real world, and makes it exciting, but we will still need text. Reading is key 


to understanding what we see with VR. 


Cynthia Haynes 


We must become the wall upon which all manner of inscriptions (texts) live alongside each 


other. Text is alive. 


Peter Wasilko 


Text is the most expressive control medium. 
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Barbara Tversky 


There are many routes to the human mind, alone and together, sight, sound, smell, touch, 
proprioception, taste, each with its own uniqueness and richness. The mind can savor each 
one and can imagine one from another, the movie that runs through the mind reading a novel, 
the floating images evoked by poetry, the ecstasy from music. 

The enveloping presence that VR may provide can be awesome, virtual worlds and “real” 
ones can be further enriched by AR. XR may create worlds we have yet to sense or imagine, 
worlds that may elevate and expand imagination. 


For ill or for good. 


Michael Joyce 


While not text, the Adyoc of the mystic Johannine evangelist, is also not not-text as well as 
one of the earliest instances of XR. The American poet Charles Olson situated this process of 
ex-ternalizing/tending writing from speech at the dawn of Western consciousness, speaking 
of how humans extend reality a/k/a (make meaning) together as mythology, which Olson 
understood as the way people talk about words, or “what is said [i.e., muthos] about what is 
said [i.e.,logos].” In the poem “Letter 23” of his four-volume 20th epic Maximus Poems 
Olson indicts Plato for having “allowed this divisive / thought to stand, agreeing / that 
muthos / is false. Logos / isn’t—was facts,” and instead declares “I would be an historian as 
Herodotus was, looking / for oneself / for the evidence of what is said.” Thus, for Olson, 
mythology, rather than spec-fict stories of strange gods and goddesses, was a supremely local 


and humanly grounded occupation, an extended reality. 


Denise Schmandt-Besserat 


Communication devices are of long duration. Our Latin alphabet is more than 3000 years old. 
The clay tokens invented ca. 7500 BC by the first farmers to keep records of goods were still 
an important tool in the first millennium BC Assyrian imperial administration. Their use can 
be traced over 6000 years. (See John MacGinnis, et alliae, “Artefacts of Cognition...” 
Cambridge Archaeological Journal 24:2, 289 ff.) 
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David Jay Bolter 


A now popular myth of the metaverse—that immersive virtual reality can serve for more or 
less all human interaction—seems to exclude traditional text-based communication. But it is 
worth considering XR (in particular VR) could accommodate new forms of discursive 
writing. Earlier media (papyrus roll, codex, printed book, and the 2D web page) have each 
constituted a particular writing space. Perhaps VR could constitute a space in which text, 
images, and videos can be inscribed in or on a 3D space. This would be the digital equivalent 
of the millennia-old practice of inscribing on wood or stone. In VR, however, terrestrial 
physics need not apply: space itself becomes manipulable. In an "immersive book" the 
architecture of the 3D space—the layout of text and the relationship among textual elements 
— can contribute to the argument. This suggests a kind of digital writing that is both familiar 


and new. 


Charlie Hargood 


XR offers unique opportunities in terms of both immersive experiences and new forms of 
interaction. However, we must be mindful that (like any medium) it comes with constraints 
and costs - and these costs are not merely fiscal, but spatial, comfort, and functional. We must 
be mindful as to whether the affordances it brings to an application outweigh these costs, and 
not fall into the trap of using XR as a gimmick for its own sake. While the immersive and 
interactive qualities have been shown to make meaningful contributions to cultural, 
entertainment, and education applications we are yet to see this for text or knowledge 


applications, or evidence of what the future value is in this space. 


Jonathan Finn 


People writing text on computer often print out a draft to get a better look at it. Why is the 
real version more present, and more pleasant? For VR to take over from reality for work, 
instead of just plonking a 'screen in a screen’ we may need to capture this mysterious 
ingredient. Resolution doesn’t seem to be the factor - maybe it’s being able to hold, move and 
flex the paper in your hand? 
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Johannah Rodgers 


Johannah's Graffiti 

Never forget: On Zoom, everyone is a homonym. We are all in the 

process of being “written by” the networked electronic computational 

devices we are using to communicate and interact with other humans and 

devices. Reading the Future of Text will help you to better 

understand how this is happening and why it matters for humans, for 

machines, and for their relations. 

-- Johannah Rodgers, Author, 

Engineering Language: Teaching Machines to Read and Write in the U.S. 1869 - 1969 


Johannah's Notes 

As most people know, I consider all forms of digital communication types of inscription 
practices. Zoom is an automated writing system. The thing we all need to consider is what is 
being proprietized and how this is changing human communication practices. Inscribed 
alphabetic communication has always been a multimodal practice. However, the modes of 
that practice are being reconfigured by the machines that we are now using to “write” with. 
XR is an inscribed reality composed by humans and machines. It has all of the biases of the 
past written into it but will also enable the expression of new types of critiques. One question 
is whether those critiques will 


result in any structural changes. The “platform” is the “writing 


system” and that has been privatized. New systems of notation are possible for humans across 
distances because of the participating of digital electronic calculating machine networks. 
Human and non-human communications practices are merging/ —_—_sin new ways with XR. 
As humans, we need to come to some kind of agreement about exactly what those changes 
are in order to ensure that human communication and human interests, as opposed to machine 
communication and machine interests are prioritized over the next decades. You can say that 
the 20th C. was all about prioritizing machine interests over biological interests and that the 
21st c. may very well be all about the fate of the biological interests that remain; will they be 
further “de-naturalized” or will we begin living within the natural constraints that remain? It 


should be interesting to see how these issues play out. 
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Dene Grigar 


The impetus to move beyond our daily lives and dream other realities is part of the human 
condition. Being human means to long to escape the physical world where we are bound by 
bodies, earth, and gravity. Our yearning has led us to conceptualize afterlives that transcend 
the fundamental laws of science, to hurdle through space to explore the moon, to walk in the 
near perfect vacuum of space. But more importantly, we have achieved this goal to escape 
reality through imagination—dreaming and instantiating new ways of living and being 
through storytelling, film, and games. XR is yet another medium in which to explore the 


future of new realities, textual and beyond. 


John Cayley 


Knowledge work is a category broad enough to allow it to find a commodious dwelling place 
in XR, respondent to the developing medium. Text as tool of knowledge work has specific 
characteristics and affordances which will constrain its instantiation and effective presence in 
this medium. Textual practice is a variety of language practice. It is important for our culture 
because literacy has become important. We now have a very wide range of delivery media for 
textual practice but these are heavily biased, even since the advent of computationally 
enhanced delivery media, in favor of documentary, expository, and transactional language- 
driven functions, and also by the predominance of a still highly effective technology for the 
delivery of textual practice, the book, particularly the codex. 

The codex is a literal volume, but, suggestively, with respect to the 3D graphics which render 
textual artifacts in XR, the codex both underwrites and undermines what text may or may not 
accomplish in XR. Effectively, materially, text has (need have) no 3rd dimension in XR. 
Conceptually, it always already does not in the codex. Tablet readers prove that the pages of 
books need have no thickness and only require one (rather than two) planar surfaces in the 
world of 3D graphics. And this 2D surface, for text, only requires one contrasting difference 
to allow the text to be read: colour of the text vs. background. Consequently you can have 
text in XR but, at any one time, not much. You will still have the problem of gathering it into 
a (partially hidden, ‘closed’) ‘volume’ and of giving your readers in XR the time as well as 
the spacetaffordances to read what will be, simply, text, perceived against a 3D spatial 
textured background: an XR tablet in other words. Think of this in the context of our current 
day-to-day experience of text in space: This is almost exclusively signage, including gallery 
didactics, and advertising. Can XR do more than this? Better? If it did, graphically, wouldn't 
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this simply be too much? 

XR will, as smart speakers have done, highlight the developing reconfiguration of recorded, 
composed, and generated language in aurality, by contrast with textuality as we know it. 
Current technologies collapse — problematically in my view — vocal recognition+production 
into literal text as parsed transcription. But emerging developments also give us tools and 
affordances with the potential to conduct, yes, linguistic knowledge work in aurality. In the 
midst of this reconfiguration is where we should look for new ways and means for language 


in XR and for further constitutive developments of our associated culture practices. 


Alan Laidlaw 


Note from one of our Lab meetings: ...where we put things, that kind of like how we offload 
our mind. I've been toying with a little bit and I'd love to talk more about this idea, —I've been 
calling it ‘new morphology’ of the the shape of a thought, pretending a thought has a shape. 
Not a scientific study by any stretch. 

It's more another thought experiment along the lines of if you could print an object and put 
your associations to it, or if it could be a way to track associations. 

Another example would be like, you know, you go into a board room, you're about to have a 
brainstorming session, but you bring a rock, Silly Putty, you know these other things, and you 
ask people to hold a sharp object while talking about X. Now take some grass and hold that. 
That's more of a thought prompter. 

But the other side would be, are there physical affordances that would help us with the shape 
of a thought, sort of how we would do composition if we were just writing articles. It has 
some interesting angles, but it's sort of like outside of VR because it's actually leaning in on 
the ‘holding things’. 

Further chat followed up by: Yeah. It's funny that as far as the physical object goes, that's 
difficult, but a lot of the inspiration for this exploration comes from Morpho Space by a 
scientist named Levin. 

There's also alpha fold and protein folding. And the interesting part about the protein folding 
is when the model is trying to figure out what the protein is going to be shaped like, it starts 
with a zero point. It doesn't have a canvas like we're used to thinking it's first off, it's a 
quaternions that it's using. But all of the atoms start at a sort of a singularity, right? A zero 
point. And then as the data, as it gets riddled with metadata. They start to push out and the 
kind of the shape of the protein starts to come together as it's getting further distanced. And I 
think that there's a lot that maybe foolishly, but interesting anyway to see that this is this is 
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sort of how ideas come about. Like we have these atoms that we don't even have names for 
that are parts of ideas before they even get to names. And as the associations start to cluster in 
our heads, we associate oh to my version of of book has these frictions and these attractions, 


you know. But it's not quite a bag of words either, anyway. 


Twitter Comments 


In response to the Editor’s Tweet: “Do you have any thoughts on text and knowledge work in 
VR/AR/XR which you would want to put in the book?” Listed in order the replies were 


received: 


Nova 


Text is generally so 1D and that's hard on many neurodivergent people, if we add more 
spatial dimensions to it we can make it contain more readable information :D 
https://twitter.com/technobaboo/status/1588125433 127702529? 
s=20&t=1rkN7egmDYKhS5oJK_E_ SeQ 


@JumbliVR's Idea Engine is a great place to start, using text as a primary element while still 
augmenting it with other symbols and graphics... 
https://twitter.com/technobaboo/status/1588138195945922561? 
s=20&t=ZPKjCISIpr-4ksHYS2HYAA 


Noda - Mind Map in VR 


“The future of text in VR will be dynamic and responsive. Adjusting to user intention for 
increasingly precise rendering” Eye tracking in XR opens up some interesting UX 
possibilities. Specific to adjusting the visual display in response to directed attention. Noda is 
using the feature on Meta Quest Pro to scale distant text for legibility and to inspect 
additional detail for items that are near. 
https://twitter.com/Noda_Tech/status/1588234308673642497? 
s=20&t=ZPKjCISIpr-4ksHYS2HYAA 
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Jimmy Six-DOF 


Working with 2D info in VR is a nested reflection of how we do so in real life but with the 
real time enhanced 3D infinate possibility space canvas layered to create a human centered 
feedback loop between 2D/Text as both at once an input & an output. Web Transclusion in 
3D=2D+! 

https://twitter.com/jimmy6DOF’status/158846501053 1237888? 
s=20&t=ZPKjCISIpr-4ksHYS2HYAA 


Kezza 


Real insights on effect of text presentation type, location on reading experience in VR are 
missing & how it aids accessibility. E.g.Edge-fixed or in head-fixed location if user needs to 
move within virtual environment & using RSVP reading with rich interaction possibilities. 
https://twitter.com/Kezza_PR/status/158854955475 1696899? 
s=20&t=ZPKjCISIpr-4ksHYS2HYAA 
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Conversations from the Journal 


Conversation: Adam's Experiment 
Adam Wern, Frode Hegland, Alan Laidlaw, Brandel Zachernuk 


Adam Wern: Been playing with the Library idea for a while, and can also show PDF pages 
in 3D. But that is not very useful in itself. Much more is needed than visualisation to beat 2D, 


or analog. 


1. Wern, 2022. 


Adam Wern: Here is my first Active Reading test in 3D. You can select text and bring 
phrases out, floating and movable in 3D. Hovering over snippet’s highlights where they came 


from in the text (the yellowish in the screenshot): 


2. Wern, 2022. 
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Adam Wern: Mostly eye candy. But with some imagination we can see that it would be 
really useful to read like this. There is something about the 3D that makes text “Rich” for me. 
I connect better. 

Especially with manipulation: Fiddling. Moving. Thinking with hands. 

But as you say Frode, without storing the work it’s not much. The actual saving it is where it 
is. And a system like this must really be able to work with both PDFs and HTML. Two 
incredible important formats in digital (in what is stored in them). 

Brandel has done some interesting translations of HTML text into 3D (preserving 
typographic styling). 

Frode Hegland: Yes! How shall we do that? We have a Wordpress plug-in but should go 
further... 

Adam Wern: Two main approaches (not mutually exclusive): start hanging VM ‘directly’ 
onto text — like an Author to a paragraph, or carve out a section of the document (for example 
with a custom visual-meta tag) and put VM there. A main questions is whether it should be 


formatted as BibTex, JSON or markup when in HTML-land. 


Meanwhile, here is FoT vol | in 3D. Testing larger amounts of text. 


3. Wern, 2022. 


Sat 


Adam Wern: OK, it looks like we can fly through half a million characters on screen in 120 


fps in the browser without a problem. Clickable characters. Good to know :) 

Alan Laidlaw: Hi all. Will be out of the woods soon. Adam, if you’re willing, I’d like to try 
to install your code on my machine in order to tinker. 

Integrating w real data is stage two, now? We’re still at faker.js stage 

Adam Wern: Yes, it’s pretty fake. Well, the data comes from a real PDF (earlier screenshot), 
and a real EPUB but it has no interface (just hardcoded file references). 

Would at least be interesting to support dragging a PDF/EPUB onto the web-app and open it, 
and save the montage to file after you are done (to be opened again) 

Frode your export/save-to-Wordpress option in Author, does it export HTML, or how does it 
work? 

Color coding and seeing lengths of articles feels nice, and it kind of forms the navigation 


equivalent of a ragged margin for better memorability (if the view stay constant) 


4. Wern, 2022. 


Adam Wern: The actual screenshots is just material for discussion. I’m very aware that 
showing lots 3D texts can look cool, but it maybe useless for doing anything more 
meaningful. So feel free to shoot down bad things, everyone! 

One thing that I do like is the seamless transition between overview and detailed view of 
texts. Basically a ZUI 

Frode Hegland: Author posts to Wordpress but it’s broken at the moment, please tell me 
what it should do, for our purposes and I’ll see what I can do. This is all very exciting! 
Brandel Zachernuk: I really liked what Mike Alger said about text, backing and contrast - 
and there was a good talk at Google IO a few years ago where they introduced ‘Distance- 
Independent Millimeters’ (‘DMM’) as a perceptual unit for describing sizes too: https://t.co/ 
3VrvoN6v1L 

Adam Wern: It’s interesting — with text in 3D on a regular screen (like the above prototypes) 


it feels natural to ‘just pinch’ to get a nice zoom level for your particular eye-sight and type of 
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reading. It’s effectively moving your body (camera view) there, but in VR that movement 
feels like a more brutal thing, as it affect your sense ‘gravity’, peripheral vision, and has 
implications for nausea, etc. So something like DMMs, perhaps adjusted for your vision like 
your default browser font-size, will be much more important in VR than ‘flat’ 3D. 

Brandel Zachernuk: Absolutely—there are implications from the stereo-parallax aspect of 
size as well as the perceptual degrees of arc. There are also aspects of technical 
implementation for display—small-and-close text will feel different because of the eye-strain 
caused trying to focus on it. 

But, small-and-close feels more different than you would expect to large-and-distant, 
especially in 6DoF. 

Adam Wern: Contrast will also be interesting, especially for AR. Colourful text with poor 
contrast or eye-sore combinations (like my things above ;) is more useful for categorising & 
scanning than reading. In XR we'll have to use blur, font outlines, and semi-transparent 
materials to do effective floating text — as the background can be anything from real life 
Brandel Zachernuk: Yes I can definitely expect that real-world detail will get in the way 
pretty badly. 

It’s fantastic that Troika actually creates “Signed Distance Field” type - it’s one thing I didn’t 
succeed in implementing in the rich format, I wonder if it would be possible to supply my 
dom-to-three stuff as encouragement to support more complex hierarchies of type. 

SDF retains crispness at small sizes without jagged aliasing effects, and surprisingly smooth 
contours from a relatively small source texture size—Valve introduced it here: 
https://steamcdn-a.akamaihd.net/apps/valve/2007/ 
SIGGRAPH2007_AlphaTestedMagnification.pdf 

Adam Wern: SDF is nice, but not perfect. Sharp edges are slightly 'tapered'. So it’s more 
round than ideal 

Fonts look friendlier :) 

But overall it feels nice, and scaled well both in quantity and zoom-level 

Brandel Zachernuk: Absolutely! I haven't peeked under the hood yet - do you know if 
Troika-text does directionally-biased SDF atlases or are they greyscale? The Valve paper 
indicates that by using multiple channels for directional biases you can dramatically increase 
the maximum detail for sharp contours 

Adam Wern: DOM to Troika would be really useful. Basic rich text - bold, headlines, links, 
and we have ourself a hypertext system that scales well - counted 500K characters until it 


started complaining. 


Haven't looked under the hood in Troika yet. 


5o3 


Brandel Zachernuk: ooh actually either spector.js or @thespite’s WebGL inspector tools 


might help (and to a lesser extent the canvas inspector in Safari). 


Adam Wern: Thanks! Will try them to figure out. SOOK was probably not a limit that can’t 
be worked around. Something exceeded some GL “texture dimension”. Troika can also be 
used with shaders that curve text, or automatically billboard things on the gpu. 

Brandel Zachernuk:Something I did and really like for timeline VR is ‘greeking’—creating 
nontransparent quads per-word in a document for display below the threshold of legibility 
based on display size. It’s effectively a level-of-detail option to alternate between in order to 
balance the value of seeing layout vs. the cost of rendering all those glyphs. Adobe InDesign 
has done it in the past for 2D, it looks like present-day Illustrator doesn’t. 


Frode Hegland: And then there is this: TikTok: https://t.co/OgiU6iRfA1 


Date Chooser Solar System. Vidovic, 2022. 


Adam Wern: That date-picker may be a joke, but the underlying idea of rotations for 
scrubbing time is very solid. And virtual controls have the advantage over a regular rotational 
knobs in that you can move outwards from the centre to get more fine-grained control. ve 
missed that in video scrubbing many times: scrubbing roughly first - and then going to 


specific frame with precision. Regular sliders don’t cut it. 


Frode Hegland: Yes, a joke, but a thought provoking one. 
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Adam Wern: On interfaces, I wonder if something looking like this would work for voice 
recognition, surfacing alternative interpretations for a phrase. To me, being misunderstood by 


voice recognition feels so irritating that I never use it. Fixing mistakes should be much easier. 


https://twitter.com/azlenelza/status/1331623011049500678 


Threads Interface 


Sentence Suances 


Paragraph 


Threads Interface. Elza, 2020. 


Conversation: Experiments with Bob Horn Mural 


Brandel Zachernuk, Frode Hegland, Adam Wern, Brendan Langen 


Frode Hegland: Yesterday, 11th of February we had a regular Friday meeting where we were 
joined by Fabien Benetou and semi-regular, now more regular, Bob Horn. Because of Bob’s 
work with murals, we spent some time going through the basics of what a mural could be in 
AR and VR, so Brandel built the following. The dialog below is from our discussion on 
Twitter. The video is quite hard to watch because of the constant movement, which is a great 
example of the power of VR: For Brandel this was a completely smooth experience and we 
really should experience it in VR ourselves. I have put up a link to the VR version in our 
blog, so that when you are in VR you can simply go to our page and easily access this. It is in 


the VR Resources Category: 


https://futuretextlab.info/category/vr-resource/ 


Chat log is on our blog, as usual: https://futuretextlab.info/2022/02/11/chat-11feb-2022/ 
Video of full meeting: https://youtu.be/Oh8yDKtPXD8 


Efe 


Transcript will be up in this category when done: 


https://futuretextlab.info/category/transcript/ 


Brandel’s Mural 


Bob Horn Mural. Zachernuk, 2022. 


Brandel Zachernuk: I dropped a static export of the mural in here: https://t.co/jH26I9JFTY 
Video walkthrough : https://t.co/jH26I9JFTY with transcript: 

“The NIREX poster in WebXR right now, it’s just a series of 2048 by 2048 rectangles and the 
end as well. But it’s nice, you know, it’s big and we can kind of navigate around it. I have 
this. Navigation is non-linear, so that small movements are small, but big movements result 
in big translations and sort of it’s proportionate to the square of the magnitude of the original 
motion so that we have the ability to get from one side of it to another without losing that fine 
detail. But now I’m zeroing out the vertical translation for the most part. This is kind of 
navigable with my hand at that this height. But it’s interesting. It’s really cool to be able to 
have these views of it and to be able to appreciate it at the size at which it’s sort of intended 
to be viewed at. Yeah, I’m pretty interested in it, and if necessary, obviously this information 
here is giving it the limits of its readability based on this particular set of pages that I’ve 
exported. But it, if necessary or possible, you can increase the resolution of this double edit or 
more or make use of some kind of adaptive display. I’m not aware of a specific PDF or at this 
point that would be able to pull this in natively, but a little bit of working. That’s definitely 
possible. Yeah, I like it. I also like this nonlinear thing. This is something that I’ve kind of 


made use of quite a bit in in my own work is having something that always has some action. 
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But given that we only have a certain arm reach range, being able to kind of pinch here and 
then throw this way back. It’s really useful. These are one meter by one meter squares on the 
ground, and I don’t have arms that long, but it means that we are able to relatively fluidly and 
effortlessly. And if? Get into these different kind of vantage points without having to have 
strict changes in modality. So, yeah. Hmm. I think it’s an interesting thing to play with, and I 
look forward to making use of more data for this kind of visualisation in the future.” 

oh, it only works with the left hand - I am left handed and also inconsiderate 

Adam Wern: Nice - and really like the non-linear navigation! 

Could be coupled with gestural ‘modifiers’ so it’s turned on by when needed. (like sticking 
out a pinky). 

The whole idea of mural in VR is interesting. A perfect fit for the really big posters. 

And where the depth dimension fits in. Could imagine stepping through a region to get more 
information. 

Or that the some labels like headings and dates stick out like tabs when the mural is viewed 
from a sharp angle. Or that the floor doubles as that timeline. 

An audio guide with moving hands for the actual mural would be nice. The place of the 
listener can be further back and floating hands can be expressive near the material without a 
body covering the view. Can be sectioned like a museum guide with numbers and indicated 


by floating markers. 
Adam Mural with Extracted Dates 


Adam Wern: Brandel, Here are dates dynamically extracted from the PDF-text through 


pdf.js (which also renders the texture via a canvas) and added as rotated text tabs. Imagine 


searching by voice and tabs pop out with results & ‘ 


So7 
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Extracted Dates. Wern, 2022. 


Brendan Langen: Yes! very good to see the data we can pull out from this. 

Brandel Zachernuk: Oooh excellent! I got recording voice in places in VR working last 
night, it’s at 

https://zachernuk.neocities.org/2022/audio-record/ 

Adam Wern: Other ideas: folding out large font-size text (probably headings) so that you 


can see headings from far away. 

Could be added to his mural, and is would be very interesting with voice search or Named 
Entity Recognition (NER). 

220 AD BCE Chinese Zhuanshu completed simplified to Lishu (Clerical script) 


Conversation: USD (Universal Scene Description) 
Brandel Zachernuk, Adam Wern 


This is from the creator of the USD format, ex-Pixar, now-Adobe: 

https://youtu.be/FAY 39CUEKpE 

“T keep thinking about - when I think about the metaverse, I keep imagining there is a 
metaverse browser, that is, you send links and the USD is the HTML of it that gives you the - 
it’s not a web page, it’s a web space now. And so then this browser, you know, it’s a browser 


that on desktop looks like a browser but in VR—you get in there and so that is more 
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immersive but conceptually has the same model. And so there’s almost like a JavaScript or 
something that on top of it that gives you that execution and things that can happen based on 


triggering from events and things like that.” 


Guido Quaroni 


Oh, the full, tidied transcript is up here too: 
https://t.co/qgVzD3PEIwu or 
https://cestum.com/open-metaverse-podcast/the-genesis-of-usd/ 


https://en.wikipedia.org/wiki/Universal_Scene_Description for wikipedia definition. 


Adam Wern: I’m glad they seem very aware of the limits of declarative formats for things 
like animation, UI behavior, physics, etc. Looking at SVG, CSS, HTML, SwiftUI, and tons of 
other (mostly) declarative formats we always seem to need to bypass rigid default behaviours 
for doing anything ambitious. And the expressiveness of imperative code has proven 
unmatched. 

Ambitious interactive 3D would be more like and app - a world rather than a model. And 
JavaScript is the main switch board for very dynamic things like webapps, while HTML 
becomes more of an empty shell. In that sense I would rather go to an index.js directly (or 
JavaScript baked into the USD), and skip the double declaration of HTML and JavaScript. 

On the other hand: an index.html can act as a loading screen, fallback page, manifest file, and 


metadata wrapper — functions which should be filled anyway. 
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History of Text Timeline 


This timeline is included partly as a resource to look through, but also as an extra data point 
for when you might look up text in the document with a Find command and the associated 
dates will then also appear. 

We understand that this will never be, nor aims to be, a complete and accurate history 
of text. There will be errors in omission, facts and dates will only be solid for the most recent 
events. The timeline format is ill suited for non-sharply delineated periods of time so we have 
tried to address that with language, such as liberal use of ‘ca’ and date ranges. The history of 
ideas is especially fraught and there will be issues we have not even thought about. What this 
aims to be however, is a useful guide for at least some of the major events and sequences 
which has brought us where we are, and which may help guide us to where we want to be 
with text. Since the format is so simple we aim that it should at least be useful to students to 
get a lay of the temporal land. 

In the Future Text Lab we are looking at how to incorporate timeline information into 
concepts so this timeline is also available as a JOSN file. 

For any suggestions or issues, please email the editor Frode Alexander Hegland at 
frode@hegland.com and you will be credited as a Contributor in the next Edition. It would be 
great if you could use this format: Year (even if you have to use ‘ca’ or other terms) Event/ 


thing by person at organisation (if applicable) 
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13,8 Billion Years Ago 


ca 13,800,000,000 years ago the universe comes into being. There was no ‘instant’ of 
creation. The universe didn’t flash into existence, it came into being as an all-encompassing, 
interactive, quantum wave. There is no going back. From pure energy to all there is today, the 
universe gets more complicated and more interactive one Planck moment at a time 

ca 4,540,000,000 years ago the earth and the solar system is formed 


ca 4,400,000,000 years ago oceans form, providing a substrate for life with rich potential for 


interactions 


Let’s pause before we continue the journey into the next, great step (that of life itself ). Look 
at these dates - the solar system has been around for roughly 1 of the universe’s existence. 
That is something to marvel at. It’s easy to imagine vast intergalactic civilizations having 
come and gone over the life of the universe, but it turns out that there actually isn’t that much 
time in the past. We’re pretty early inhabitants. There may have been one generation of stars 
similar to our own before us —maximum. So, maybe there hasn’t been enough time for 
advanced civilizations to evolve. That we might be one of the most advanced consciousness 


in creation (or perhaps the only one) is a sobering thought. Can we handle this responsibility? 


ca 4,000,000,000 years ago Self-replicating molecules appear. Life is happening. It’s pretty 
basic, but it’s happening 

ca 3,500,000,000 years ago Single-celled organisms 

ca 3,000,000,000 years ago Viruses, though they may be much older 

ca 580,000,000 years ago Complex multicellular life 

ca 250,000,000 years ago or less—it is hard to be sure, DNA, with complex ‘letters’ of 


interaction takes life to a whole new level 


250 Million-3,6 Million 


2,7-2,5, 1,9-1,7 and 1,1-0.9 million years ago, the earth sees rapid climate change (on the 
scale of lifetimes of individuals, not species) spurring on hominid evolution in the Rift Valley 
in Africa, with each period coinciding with brain development. During the period 1,9-1,7 the 
number of hominid species reached its peak and Homo Erectus appeared. Tool development 


also coincided with these cycles of rapid climate change, including Oldowan, Acheulean and 
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Mousterian. For more on this topic, and how the planet shaped us in general refer to Origins 
by Lewis Dartnell 

ca 3,600,000 years ago Our ancestors walk upright and they loose body hair 

ca 2,300,000 years ago Homo Habilis, the tool user, is our oldest ancestor to use tools 


ca 2,000,000 years ago Olduwan tool Culture begins. Its key feature was the method of 


chipping stones to create a chopping or cutting edge. 


2,000,000-50,000 BCE 


ca 500,000 years ago Earliest evidence of purpose-built shelters. Found near Chichibu, Japan 
ca 400,000 years ago Early humans begin to hunt with spears 

ca 280,000 years ago First complex stone blades and grinding stones 

ca 150,000 years ago Humans possibly capable of speech 

ca 100,000-200,000 Modern Humans 


50,000-3,000 BCE 


ca 50000 BCE Our ‘Great leap forward’. Human culture starts to change more rapidly 
(burying our dead ritually, clothes from animal hides, complex hunting techniques) 

ca 44000 BCE Oldest known cave painting, found in the Franco-Cantabrian region in 
western Europe and Sulawesi, Indonesia 

ca 35400 BCE Oldest-known example of figurative art, in Sulawesi, Indonesia 

ca 11000 BCE Cave art by young children in the Rouffignac Cave 

ca 7500 BCE Near Eastern counters ‘Tokens’ to keep track of goods are the earliest known 
antecedents of the Mesopotamian Cuneiform script 

ca 6600 BCE Eleven isolated symbols carved on tortoise shells were found at Jiahu, an 
archaeological site in the Henan province of China, some bearing a striking resemblance to 
certain modern characters but the connection is not established 

ca 4500 BCE Proto-Indo-European language developed, probably somewhere near the Black 
Sea, and probably spreading because its speakers invented horse riding. Today 60% of 
modern humans speak a daughter language, 27% as their mother tongue 

ca 4000 BCE Possible preliterate images which may have been symbols (such as Gerzean 


pottery) which could have been precursors to Egyptian hieroglyphic writing 
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4000 BCE 


ca 3500 BCE Egyptian Proto-hieroglyphic symbol systems 


ca 3300 BCE Reduction of three-dimensional Near Eastern tokens into two-dimensional 


signs on envelopes holding tokens 


ca 3200 BCE First logographic Near Eastern accounting lists written on clay tablets by 


impressing tokens 


ca 3100 BCE First logographic proto-cuneiform signs traced with a stylus on accounting 
tablets 


ca 3000 BCE First proto-cuneiform phonetic signs to represent personal names on economic 
tablets 


ca 3000 BCE First known use of papyrus for writing. Previously Egyptians had been writing 


on stone and pottery 


ca 3000-1000 BCE Hieratic (‘priestly') cursive writing system used for Egyptian until the 


rise of Demotic. Primarily written in ink with a reed pen on papyrus. 


3000 BCE 


2900 BCE First known air mail. Egyptian sailors released carrier pigeons from ships to pre- 


announce their arrival 


ca 2800 BCE First full sentence written in mature Egyptian hieroglyphs so far discovered. 


Found on a seal impression in the tomb of Seth-Peribsen at Umm el-Qa'ab 

ca 2700 BCE First cuneiform texts which departs from accounting: funerary texts 
ca 2600 BCE Sumerian language develops 

ca 2600 BCE Egyptian language develops 

ca 2400 BCE Akkadian language develops 

ca 2400 BCE First cuneiform tablet dealing with trade 


ca 2300 BCE First written sentences. These texts were inscribed on worshippers’ votive 


statues dedicated to a god and requesting immortality 
ca 2300 BCE First named author, Enheduanna, daughter of Sargon the Great 


ca 2300 BCE Oldest known dictionaries of cuneiform tablets with bilingual Sumerian— 


Akkadian wordlists, discovered in Ebla (modern Syria) 
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ca 2000 BCE Classical period of the Sumerian Cuneiform Script 
ca 2000 BCE First known library catalog in the Sumerian city of Nippur 


ca 2000 BCE Abacus (from Greek meaning “board strewn with sand or dust used for 
drawing geometric figures or calculating”), the first known calculator, is invented in 


Babylonia (Iraq) 
ca 2100 BCE Elamite language develops 
ca 2100-1500 BCE Proto-Sinaitic script, the earliest trace of alphabetic writing known, in 


the Egyptian Pharaoh’s turquoise mines at Serabit el-Khadim in the Sinai Peninsula 


2000 BCE 


ca 1900 BCE First known cipher (not yet decoded), in tomb of Khnumhotep II 

ca 1750 BCE Hammurabi’s Code, by Hammurabi, ruler of Babylon 

ca 1700 BCE Hittite language develops 

ca 1600 BCE Earliest known medical document, the Edwin Smith Medical Papyrus, thought 


based on material from 3000 BCE, including the first reference to the human brain 


ca 1500 BCE Phoenician alphabet of 22 consonants was among the early mature alphabets. It 
spread over the Mediterranean and led to the Greek, Hebrew, Roman, Arabic and modern 


alphabets 
ca 1450 BCE Greek language develops 
ca 1500 BCE Earliest book known, the Ebers papyrus, a 20 meter scroll 


ca 1500 BCE First known use of movable type (stamps reused to repeat symbols identically), 
the Phaistos Disc, and first font 


ca 1300 BCE First known inclusion of words on a map, in Mesopotamia 


ca. 1300-1190 BCE The Ugaritic writing system a cuneiform augmented abjad (consonantal 


alphabet) for Ugaritic, an extinct Northwest Semitic language 


ca 1300s BCE Wax tablet with stylus: origins are uncertain but known to have been used at 


least until the 1860s CE, for example in the fish market in Rouen, France 
1200s BCE Late Bronze Age collapse 
ca 1250-1192 BCE Earliest confirmed evidence of Chinese script, Oracle bones script 


ca 1200 BCE Torah was copied onto a scroll by Moses according to the Hebrew tradition 
(date disputed) 
ca 1200 BCE Old Chinese language develops 
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ca 1100 BC-256 BCE Chinese Jinwen (Bronzeware Script) 
1000-300 BCE Chinese bronze inscriptions/script 

1000s BCE the Gezer Calendar, first vertically-formatted list 
ca 1000 BCE Hebrew language develops 


1000 BCE 


1000 BCE Chinese Seal script evolved organically out of the bronze script 
900-400 BCE The Greek Alphabet emerged around the ninth or eight century BCE which 


had distinct letters for vowels, not only consonants. Many versions of the Greek alphabet 
existed but by the fourth century it had been standardised into twenty-four letters, ordered 


from alpha to omega 

ca 700 BCE Latin language develops 

700s BCE Alphabetic writing entered the Greek world from the Levant 

650 BCE Demotic Egyptian script following Late Egyptian and preceding Coptic. The term 
was first used by the Greek historian Herodotus to distinguish it from hieratic and 
hieroglyphic scripts 

500s First known curated museum. Mesopotamian artifacts spanning 1,500 years, by Princess 
Ennigaldi, daughter of King Nabonidus 

ca 500 BCE Sanskrit language develops 


ca 550 BCE First official mail service, by Cyrus the Great, stretching from Post, Iran to 
Hakha, Myanmar 


ca 500 BCE Astadhyayt by Panini, quasi-generative grammar of Sanskrit, anticipating 
Chomsky 

300s BCE The basic form of the Codex invented in Pergamon 

ca 300 BCE Tamil language develops 

300s BCE Reed pens for writing on papyrus 

310/305—240 BCE The Pinakes, the first library catalog at the Library of Alexandria 
285-246 BCE Alexandria founded by Alexander the Great 

283 BCE Library of Alexandria founded by Ptolemy I and II 

257-180 BCE Punctuation is invented at the Library of Alexandria by Aristophanes of 


Byzantium 


256-206 BCE Chinese Zhuanshu (Seal Script). 
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206 BCE Chinese Zhuanshu starts being simplified to Lishu (Clerical script) 250 Parchment 
Scrolls 


ca 230 BCE The letter ‘G’, by Spurius Carvilius Ruga, the first known inventor of a letter 
200s BCE Quill used until about the 19th century CE, when replaced by the pen 


200s BCE Alphabetization developed, probably in Alexandria by Callimachus to catalog the 
Great Library 


200s BCE Erya, first known dictionary 
ca 131-59 BCE BCEActa diurna, daily news by government, published in Rome 
179-141 BCE Earliest extant paper fragment in Fangmatan in Gansu province, China 


before 134 BCE First character encoding, by Cleoxenus and Democleitus, described by 


Polybius. Each Greek letter was converted to 2 digits (1 to 5), then to smoke or fire signals 
63 BCE & ‘ampersand’ proposed by Marcus Tiro 


ca 55 BCE The book in the form of folded sheets, not just a stack of sheets, by Julius Caesar, 
in his reports on the Gallic Wars 
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OCE 


ca 50 Earliest surviving example of Old Roman Cursive script: a speech by Claudius 
79 Earliest tables of contents by Pliny the Elder in Naturalis Historia (Natural History) 
79 Earliest known marketing pun and portmanteau word: wine jars in Pompeii marked 
“Vesuvinum’ (Vesuvius wine) 

79 Two SATOR AREPO word squares in Pompeii, perhaps with Christian associations, 


making them the earliest surviving Christian inscriptions 


100 CE 


200 


ca 200 New Roman (or Minuscule) Cursive script which evolved into modern lower case 


letterforms 
ca 220 Earliest surviving woodblock printed fragment (China) 


220 Chinese Zhuanshu completed simplified to Lishu (Clerical script) 


300 


ca 300 Maya writing 

ca 300 Latin handwriting starts to use larger letters at the start of sentences, though the same 
shape (not mixed case) 

330-360 Codex Sinaiticus, the oldest extant codex; a biblical manuscript written in Greek 


367 Old Roman Cursive script banned except for official imperial documents, eventually 
leading to lower case text (derived from New Roman Cursive) being normal and upper case 


exceptional 
400 


420-589 Chinese Kaishu script (Regular Script) replaces Lishu 
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400s Demotic Egyptian script dies out from active use 


500 


Before 500s Literacy introduced to Japan in the form of the Chinese writing system, via 
Baekje 

500-1000 Florilegium, which are selections of ‘flowers’(select passages) from work, rather 
than a summary, to help people deal with the volume of books 


593 Woodblock printing starts in China 


600 


600s Quill pens, made from the outer feathers of crows and other large birds, becomes 


popular 


700 


ca 700s Word spacing pioneered by Celtic monks 
ca 700 St Cuthbert Gospel, the oldest surviving Western book, which still has its original 


goatskin leather cover 
700s Japanese writing develops away from Chinese 


764 Empress Koken commissions the earliest known examples of woodblock printing in 


Japan 


800 


800s paper starts to replace parchment as the primary writing material for administrative uses 
in Baghdad 

813 Council of Tours decreed sermons should be in vulgar language not Latin. This may have 
triggered early Romance languages to be spelt literally, rather than as Latin with distorted 
pronunciation 

842 Oaths of Strasbourg, first surviving document in Romance (early French), with parallel 


version in Frankish (early Germanic) 
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868 The oldest known printed book, The Diamond Sutra, a Buddhist book in China 


854-931 Prototype professional peer-review process recommended in the Ethics of the 
Physician written by Ishaq ibn ‘Alt al-Ruhawi 


900 


ca 900 Screen Printing in China during the Song Dynasty 


900s Arabic numerals come to Spain, though they were not commonly used until the 
fourteenth century. 


960-1279 Chinese Kaishu script evolves to Songti script 


1000 


1080 The Missal of Silos, the oldest known document on paper created in Europe 


1056 First recorded paper mill in Xativa on the Iberian Peninsula 


1100 


1190 First paper mill in France 


1200 


1200s The term “Originalia’ is coined in contrast to Florilegia, indicating a greater authority 


to original sources than excerpts 


1246 Call numbers associated with the location of books, in the Library at Amiens Cathedral 


in France 
1276 Paper mills established in Italy 
1290 Ars Magna by Ramon Lul 


1300 


1377 Jikji the oldest surviving book printed using moveable metal type by Gyeonghan in 
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Korea 


1300s The word ‘history’ meant, “relation of incidents whether true of false.” The word goes 
back to the Proto-Indo-European root of wid-tor weid, it literally mean “to know” and “to 
see.” 

1304-1374 Humanism founded by Francesco Petrarch, reviving enthusiasm for ancient 
Roman thinkers, with books as the centre of their discourse 

1320 First paper mills in Germany 

1340-1350 First paper mills in Holland 


1346 First known two-color print, a frontispiece to a Buddhist sutra scroll 


1400 


1400s First prototype of a Jacquard-type loom by Jean le Calabrais 

1424 The University of Cambridge has one of the largest libraries in Europe with just 122 
books. Books are still handwritten on parchment 

1453 Constantinople captured by the Turks and books from its Imperial Library are burned or 
removed, marking the end of the last of the great libraries of the ancient world 

1455 ‘Gutenberg Bible’, also-called Forty-two-line Bible, or Mazarin Bible, the first 
complete moveable type printed book extant in the West, printed by Johannes Gutenberg 
1457 First known color printing is used in Mainz Psalter by Johann Faust and his son-in-law 
Peter Schoffer 

1470 Roman typeface, the first recognisably modern typeface, a combination of capital letters 
inspired by ancient Roman architectural inscriptions and Carolingian minuscules, developed 
by Nicolas Jenson 

1470 First printed joke book, Facetiae by Poggio Bracciolini 

1470 Earliest extant example of sequential numbering in a book, Sermo in festo 
praesentationis beatissimae Mariae virginis, printed in Cologne. This did not become 
standard for another half century. Peter Schoffér, apprentice of Gutenberg, is the inventor of 
the title page and Arnold Therhoernen in Cologne, is one of the first to use both a title page 
and page numbers 

Late 1470s, title, author, and publisher information included by printers on the first inside 
page of a book 

1479 Manicule in Breviarium totius juris canonici, compiled by Paolo Attavanti printed in 


Milan by the German firm of Leonhard Pachel and Ulrich Scinzenzeller 
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1481 First marginal annotations used in printed texts on a Venetian edition of Horace with 


commentaries by Acro and Porphyry 

1483 First Talmud printed 

End of the 1400s almost all printed books have title pages 

End of the 1400s the numerals 4, 5, and 7 begin to take the forms we are familiar with today 


1500 


1500-1700 Handwritten newsletters in Europe called avvisi, reporti, gazzette, ragguagli, 


nouvelles, advis, corantos, courantes and Zeitungen 


1500s Garamond typeface. Claude Garamont, a French type designer, publisher and punch- 
cutter lived in Paris. Thus, many old-style serif typefaces are collectively known by his name 


as ‘Garamond’ 


1500s The word ‘history’ is differentiated into ‘history’ and ’story’ in English, though in other 


languages, such as Spanish and Norwegian there is still no distinction 

1500s Maya writing mostly fallen out of use 

ca1500 Etching for printing by Daniel Hopfer 

1501 Italic typeface by Aldus Manutius 

1513 Likely first pagination with Arabic numerals in Cornucopiae by Niccoloo Perotti 


1517 Martin Luther posts a thesis against indulgences and thus sparking what would be 
called the Reformation, a questioning of authority which would spur greater literacy rates and 


interest in education 

1530s Monasteries disolved in England 

1538 Latin-English wordbook by Sir Thomas Elyot 

1539 Henry the Eighth’s Great Bible, by Myles Coverdale banning all glossing 


1540 Henry the Eighth’s authorised Grammar, of which formed the basis of schoolbooks in 
England for the next 300 years 


1545 Bibliotheca universalis by Conrad Gessner, a complete bibliography of all printed 
books (except itself) 


1556 Notizie Scritte, first monthly newspaper published in Venice 


1557 The Geneva Bible, the primary Bible of 16th-century English Protestantism displaces 
the Great Bible 


1560 First blueprints for the modern, wood-encased carpentry pencil by Simonio and 
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Lyndiana Bernacotti 


1564 Graphite for pencils comes into widespread use following the discovery of a large 


graphite deposit in Borrowdale, England 


1568 Bishops' Bible, English translation of the Bible produced under the authority of the 
established Church of England and later used as the base text for the King James Bible 


1575 First paper mills in Mexico 

1565 Mechanical/Lead holder pencil by Conrad Gesner 

1588 First commercially successful paper mill in Britain by John Spilman in Kent 
1593 Index to content in a book, by Christopher Marlowe in Hero and Leander 


1595 The first printed catalog of an institutional library, the Nomenclator of Leiden 
University Library 


1600 


1600 Orbis Sensualium Pictus textbook for children by John Amos Comenius 


1604 Relation aller Fiirnemmen und gedenckwiirdigen Historien, first weekly newspaper, 


published in Germany by Johann Carolus 

1611 King James Bible 

1642 Mezzotint Printmaking by Ludwig von Siegen 

1648 Part emoticon ‘(smiling yet:)’ by poet Robert Herrick 

1665 Journal des scavans, in Paris, first academic journal 

1665 Philosophical Transactions of The Royal Society, in London, second academic journal 
1665 Oxford Gazette, first English newspaper 

1667 Acoustic string telephone by Robert Hooke 

1674 First decipherment of a script, the Staveless Runes, by Magnus Celsius 
1677 Artificial versifying by John Peter 

1600s Quills become more pointed and flexible 

1690 First paper mills in the USA 


1700 


1702 The Daily Courant, the world's first daily newspaper, printed on paper so cheap it was 


designed to be thrown away after reading 
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1704 Daniel Defoe, considered the first journalist, publishes The Review 

1704 Newton’s Opticks, the first major scientific book published in English, not Latin 
1706 Newton’s Opticks translated into Latin 

1714 First patent for a mechanical typewriter issued to Henry Mill 


1723 De Etruria regali libri VII Thomas Dempster used sans serif typeface to represent 


inscriptions in Ancient Greek and Etruscan 


1725 Improvement to the Jacquard-type loom by Basile Bouchon who introduced the 
principle of using a perforated band of paper 

1731 First peer-reviewed journal, Medical Essays and Observations (Philosophical Society of 
Edinburgh, Edinburgh). 

1739 Last international treaty written in Latin, the Treaty of Belgrade, indicating the new pre- 
eminence of living languages over dead ones 


1748 First modern use of sans-serif (“grotesque”) lettering, anonymous letter carver, grotto at 
Stourhead, England 


1755 A Dictionary of the English Language by Samuel Johnson 
1767 Index Card organization by Carl Linnaeus 


1769 Every house in Britain needs to have a number for addressing, introduced with the 
Stamp Act 

1770 Natural rubber used as an eraser by Edward Nairne 

1771 UK Parliament formally gives journalists the right to report proceedings 

1772 Aquatint printing by Peter Perez Burdett, named by Paul Sandby 

1780 Didot and Bodoni by Firmin Didot and Giambattista Bodoni, the first modern Roman 
typefaces 

1780 First card catalog by librarian Gottfried van Swieten, Prefect of the Imperial Library, 


Austria 

1783 James Madison of Virginia proposes the creation of a congressional library 

1786 Rounded sans-serif script font developed by Valentin Hatiy for the use of the blind to 
read with their fingers 

1787 Constitution of the United States, mentioned here as a milestone in written documents 
producing and framing a society 


1787 The Federalist Papers by Alexander Hamilton with John Jay and James Madison in The 
Independent Journal, considered the most important documents for interpreting and 


understanding the original intent of the Constitution of the United States 
1791 First card catalog for libraries, using the back of playing cards by a group of men with 
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bibliographic experience led by Barthélemy Mercier 
1795 Modern Pencil by Nicholas-Jacques Conté 
1796 Lithography by Alois Senefelder 

1796 Colour Lithography by Alois Senefelder 


1799 The Fourdrinier machine, a continuous paper making machine by Louis-Nicolas Robert 


of France 


1800 


1800 The Library of Congress established when President John Adams signed an act of 
Congress also providing for the transfer of the seat of government from Philadelphia to the 


new capital city of Washington 

1801 Blackboard by James Pillans 

1801 Carbon Paper by Pellegrino Turri 

1804 Jacquard loom by Joseph Marie Jacquard 
1806 Patent for Carbon Paper by Ralph Wedgwood 
1875 First literary agents 


1810 


1816 First typeface without serifs by William Caslon IV 
1816 First working Telegraph by Francis Ronalds used static electricity; it was rejected by the 


Admiralty as “wholly unnecessary” 


1817 A Code of Signals for the Merchant Service, the first general system of signalling for 


merchant vessels by Captain Frederick Marryat 


1819 Rotary printing press by David Napier 


1820 


1822 Mechanical Pencil with a ‘Mechanism to Propel Replaceable Lead’ by Sampson 


Mordan and John Isaac Hawkins 
1828 Pencil Sharpener by Bernard Lassimonne 
1829 Embossed printing invented by Louis Braille 
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1830 


1836 Chorded Keyboard by Wheatstone and Cooke 

1837 Early forerunner of Morse Code by Samuel F. B. Morse, Joseph Henry, and Alfred Vail 
1839 Vulcanized rubber used for erasers by Charles Goodyear 

1839 Electrical Telegraph commercialised by Sir William Fothergill Cooke 


1840 


1843 Rotary Drum Printing by Richard March Hoe 
1843 Wood pulp introduced to paper mills for paper production 
1844 Newsprint by Charles Fenerty of Canada. Designed for use in printing presses that 


employ a long web (continuous sheet) of paper rather than individual sheets of paper 
1844 Morse Code by Samuel F. B. Morse, Joseph Henry, and Alfred Vail, in use 
1846 Printed Output envisioned by Charles Babbage from his Difference Engine 2 


1850 


1854 Boolean algebra the mathematical basis of digital computing, developed by George 
Boole in The Laws of Thought 


1855 International Code of Signals drafted by the British Board of Trade 
1857 International Code of Signals published as the Commercial Code 


1857 National Telegraphic Review and Operators Guide lists emoticon precursors <3 and :* 


as shorthand for ‘love and kisses’ 


1857 Study On Some Deficiencies in our English Dictionaries, which identified seven 
distinct shortcomings in contemporary dictionaries published by the Unregistered Words 
Committee of The Philological Society, a small group of intellectuals in London headed by 
Richard Chenevix Trench 


1858 Eraser on pencil by Hymen Lipman 
1858 First transatlantic telegraph cable laid by Cyrus West Field 


1860 


S15 


1860s The first card catalog, designed for readers, rather than staff, by Ezra Abbott, Harvard’s 
assistant librarian 


1860 Herbert Coleridge succeeds Richard Chenevix Trench as the first editor of the 
Unregistered Words Committee’s effort; this work was the precursor of what eventually 
became the Oxford English Dictionary (OED) 


1860 Hectograph, gelatin duplicator or jellygraph printing process by Nelson S. Knaggs 
1860 The New York Herald starts the first ‘morgue’, meaning archive 
1861 The Unregistered Words Committee published the first sample pages, Herbert Coleridge 


dies and Frederick Furnivall takes over as editor 

1864 Non-Digital ‘spam’. Unsolicited group telegram advertisement 

1868 Kineograph / Flip-Book by John Barnes Linnett 

1868 The Remington by Christopher Latham Sholes, the first successful typewriter 


1870 


1870s QWERTY layout by Christopher Latham Sholes 
1874 Stencil Duplicating by Eugenio de Zuccato 
1876 Telephone patent by Alexander Graham Bell 


1876 Telephone Switch, which allowed for the formation of telephone exchanges and 
eventually networks by Tivadar Puska 

1876 Autographic Printing by Thomas Edison 

1879 The Oxford University Press agrees to publish The Unregistered Words Committee’s 
dictionary, to be edited by James Murray 

1879 Index Medicus edited by John S. Billings and Robert Fletcher, published by Frederick 
Leypoldt 


1880 


1828 On the recent Improvements in the Art of Printing published in The Quarterly Journal of 


Science, Literature, and Art, by Edward Cowper 


1850 On Printing Machines, Especially Those Used in Printing 'The Times' Newspaper 


published in Institution of Civil Engineers. Minutes of Proceedings, by Edward Cowper, 
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outlining his contribution to printing which had increased newspaper printing from 200-250 


copies per hour on a hand press to 10,000 copies per hour 

1873 First illustrated daily newspaper, The Daily Graphic, published in New York. 

1877 Current definition of entropy, by Ludwig Eduard Boltzmann 

1881 Harvard Citation Style (author date) by Edward Laurens Mark at Harvard University 


1881 Emoticon precursors as Puck magazine published a set of type-set faces expressing joy, 


melancholy, indifference and astonishment using basic type characters 
1883 Téléphonoscope concept by Albert Robida 
1884 Linotype by Ottmar Mergenthaler 


1884 The Oxford University Press agrees to publish A New English Dictionary on Historical 
Principles; Founded Mainly on the Materials Collected by The Philological Society 


1887 Snigger Point by Ambrose Bierce, a precursor emoji/emoticon symbol in the form of an 
opening parenthesis character ‘(’, but rotated 90° to the left 


1888 Ballpoint Pen by John J. Loud 


1890 


By 1890 Some papers boasted circulations of more than one million 


1890 US Census undertaken using the punched-card technology, an invention suggested by 
John S. Billings to Herman Hollerith in the company which would become IBM 

1891 Automatic Cyclostyle duplicating machine by David Gestetner 

1895 Universal Decimal Classification (UDC), starting with the Universal Bibliographic 
Repertory (RBU: Répertoire Bibliographique Universel) by Paul Otlet and Henri La Fontaine 
with the implementation being as card catalogue by Herbert Haviland Field, using the Dewey 


Decimal Classification system by Melvil Dewey 
1894 Information and Entropy in Thermodynamics by Ludwig Boltzmann 


1895 A New English Dictionary on Historical Principles renamed as the Oxford English 
Dictionary (OED) 


1900 


1901 Trans-Atlantic Radio Signal by Marconi Company 


1902 The term Diglossia coined by Karl Krumbacher to refer to the phenomenon of 


ee i, 


divergence between spoken and written language 


1903 First message to travel around the globe by Commercial Pacific Cable Company, from 
US President Theodore Roosevelt, wishing ‘“‘a happy Independence Day to the US, its 


territories and properties...” It took nine minutes for the message to travel worldwide 
1903 The Daily Mirror, the first tabloid-style newspaper 


1904 Patent for a ‘type wheel printing telegraph machine’ filed by Charles Krum which 
would go on to become Teletype in 1929 


1906—7 Photographic Copying Machines by George C. Beidler at the Rectigraph Company 
1907 Commercial Transatlantic Radio Telegraph Cable opened by Marconi Company 


1910 


1910 Felt-tip marking pen by Lee Newman 
1910’s Teleprinter, Teletext via telegraphs, by 
1910 Mundaneum by Paul Otlet and Henri La Fontaine 


1910 First criminal caught via wireless telegraph: the murderer Dr Crippen on board a 


transatlantic ship 


1913 Plantin typeface by Frank Hinman Pierpont and draughtsman Fritz Stelzer of the British 
Monotype Corporation, based on a Gros Cicero face cut in the 16th century by Robert 


Granjon 


1914 Optophone (OCR precursor) by Emanuel Goldberg, a machine which read characters 
and converted them into standard telegraph code 


1914 Handheld Scanner (OCR precursor) by Edmund Fournier d’ Albe a machine which read 


characters and converted them into tones 


1920 


1920s First full-time Type Designer Frederic Goudy 


1922 Ulysses by James Joyce, first extensive use of stream of consciousness: text conveying 
thoughts not speech 


1923 Spirit duplicator (also referred to as a Ditto machine, Banda machine, or Roneo) by 
Wilhelm Ritzerfeld 


1925 Corkboard by George Brooks 
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1926 Information in physics by Leo Szilard 

1926 research and development which would become Telex initiated by Reichspost in 
Germany 

1927 The Statistical Machine patented by Emanuel Goldberg 

1927 Futura typeface family by Paul Renner 

1924 Art Color Pencils by Faber-Castell and Caran d’Ache 

1928 Standardised punch cards by Clair D. Lake 

1929 Hellschreiber by Rudolf Hell, precursor to dot matrix printing 

1929 Practical Criticism: A Study of Literary Judgment by Richards, I.A 


1930 


1930 The Readies, a concept for portable speed reading by Bob Brown 

1931 Knowledge Machine by Emanuel Goldberg 

1931 Biro by brothers Laszl6 Biré and Gyorgy Bird 

1931 The American Telephone and Telegraph Company (AT&T) introduced its teletypewriter 


exchange service, TWX 


1932 Times New Roman typeface by Victor Lardent under the direction of Stanley Morison, 


on a commission of the Times newspaper, based on the Plantin typeface 
1932 Information in quantum and particle physics by John von Neumann 
1933 Telex by Reichspost in Germany operational 

1933 Machine translation by Petr Petrovitch Smirnov-Troyanski 


1934 Logik der Forschung by Karl R. Popper advanced the theory that the demarcation of the 


limit of scientific knowledge, is its ‘falsifiability’ and not its ‘verifiability’ 


1934 Mundaneum/ “Mondotheéque,” by Paul Otlet. Includes automated linking between “card 
catalogs with sixteen million entries, photos, documents, microfilm, and more. Work on 


integrating telegraphy and multiple media, from sound recordings to television” 
1935 Monde book by Paul Otlet 

1936 Dvorak Keyboard Layout by August Dvorak 

1937 World Brain by H. G. Wells 


1940 
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1940s-60s Information as a concept, through the works of Claude Shannon (information 
theory), Warren Weaver (machine translation), Alan Turing (universal computer), Norbert 
Wiener (cybernetics) and Friedrich Hayek (invisible hand is information) 

1942 Xerography Patent by Chester Carlson. The technique was originally called 
electrophotography 


1943 The term ‘acronym’ coined, meaning word formed from the first letters of a series of 


words 


1944 Marking pen which held ink in liquid form in its handle and used a felt tip by Walter J. 
De Groft which becomes ‘Sharpie’” in 1964 


1945 Memex proposed by Vannevar Bush in As We May Think 


1945 ENIAC first programmable, electronic, general-purpose digital computer by J. Presper 
Eckhart and John Mauchley (University of Pennsylvania) 


1946 A Logic Named Joe by Murray Leinster 
1946 Works on Machine Translation by Andrew Booth 


1947 Machine translation, suggested in a letter from Warren Weaver suggests to Norbert 


Wiener 
1946 Electric Printing Telegraph by Alexander Bain, precursor to the fax 


1948 A Mathematical Theory of Communication by Claude Shannon, including the word 
‘bit,’ short for binary digit, credited to John Tukey 


1948 The Human Use of Human Beings: Cybernetics and Society by Norbert Wiener. The 
word cybernetics was first used in the context of the study of self-governance of people by 
Plato and in 1834 by André-Marie Ampére to mean the sciences of government in his 
classification system of human knowledge. Here Norbert Wiener introduced the term for the 


scientific study of control and communication in the animal and the machine 
1949 El libro mecanico by Angela Ruiz Robles 
1949 Translation memo by Warren Weaver 


1949 The Lumitype-Photon Phototypesetting by the Photon Corporation based on the 
Lumitype of Rene Higonnet and Louis Moyroud 


1949 Fr Roberto Busa starts work on computerizing his Index Thomisticus (St Thomas 
Aquinas), in the process founding Humanities computing 


1949 The Chinese Language Character Reform Association established 


1950 


580 


ca 1950 Niklas Luhmann’s Zettelkasten system for storing and cross-referencing information 
in card indexes 

1950 Whirlwind computer at MIT including a display oscilloscope becomes operational 
1950 Computing Machinery And Intelligence by Alan Turing where he proposes the question 
‘Can machines think?’ 

1950s-60s Simplified Chinese characters created by works moderated by the government of 
the People's Republic of China 

1951 Doug Engelbart’s Epiphany: “Problems are getting more complex and urgent and have 
to be dealt with collectively — we have to deal with them collectively” 

1951 Qu’est-ce que la documentation? by Suzanne Briet 

1951 Regular expressions by mathematician Stephen Cole Kleene 

1951 Linear B deciphered as a syllabic script for early Greek, by Michael Ventris 

1951 LEO I the first general-purpose business computer, Lyons Ltd, text on paper-tape 
readers and punches 

1951 UNIVAC (UNIVersal Automatic Computer) by J. Presper Eckert and John Mauchly at 
EMCC/ Remington Rand 

1952 Manchester Mark I computer Love Letter Generator by Christopher Strachey, using a 
random number algorithm by Alan Turing 

1952 Antitrust Investigations And Trial Against IBM starts, dragging on for thirty years, 
finally being dismissed in 1982. IBM will cautiously monitor its microcomputer business 
practices, fearful of a repeat of government scrutiny 

1952-4 Dot Matrix Teletypewriter developed by Fritz Karl Preikschat 

1952 ‘Love letter generator’ aimed to generate a literary text from scratch, by Christopher 
Strachey 

1953 UNIVAC 1103 designed by Seymour Cray at the Engineering Research Associates and 
built by the Remington Rand corporation 

1953 Magic Marker by Sidney Rosenthal 

1953 The Lumitype-Photon Phototypesetting System first used to set a complete published 
book and to set a newspaper 

1954 Charactron by J. T. McNaney at Convair was a shaped electron beam cathode ray tube 
functioning both a display device and a read-only memory storing multiple characters and 
fonts on the UNIVAC 1103 

1954 IBM 740 CRT used computers to draw vector graphics images, point by point, on 35 
mm film 1956 Keyboard and Light Pen for computer text input at MIT on the Whirlwind 
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computer 

1954 The Chinese Language Character Reform Committee was founded 

1955 Teletype-setting used for newspapers 

1956 Chinese List of Simplified Characters issued by State Council 

1956 First commercial computer sold with a moving-head ‘hard disk drive’, the 305 RAMAC 
by IBM 

1956 ‘Artificial Intelligence’ term coined by John McCarthy at MIT 


1957 COMIT string processing programming language by Victor Yngve and collaborators at 
MIT 


1957 Univers typeface family by Adrian Frutiger 


1957 The term ‘initialism’ coined, a written word formed from the first letters of other words 
in a name or phrase. NATO, where the letters are sounded as a word are regarded as 


acronyms. FBI, where the letters sound as letters, are initial-words or initialisms 

1957 Dye-Sublimation printing by Noél de Plasse at Sublistatis SA 

1957 Helvetica typeface family by Max Miedinger 

1958 The Uses Of Argument by Stephen Toulmin introduces the argumentation diagram 


1958 Lisp programming language designed by John McCarthy at MIT and developed by 
Steve Russell, Timothy P. Hart, and Mike Levin 


1958 Integrated Circuit (IC) by Jack Kilby at Texas Instruments 
ca 1958 Speed reading by Evelyn Wood 


1960 


1960s ‘Word Processing’ term invented by IBM 
1960 PLATO (Programmed Logic for Automatic Teaching Operations) generalized computer- 


assisted instruction system by Donald Bitzer at the University of Illinois 


1960 Colossal Typewriter by John McCarthy and Roland Silver at Bolt, Beranek and 
Newman (BBN) 

1960 Ted Nelson’s epiphany about interactive screens becoming universal, on-line publishing 
by individuals 

1960 Suggestion for emoticon by Vladimir Nabokov 

1960 Man-Computer Symbiosis by J.C.R. Licklider at BBN 

1961 Selectric Typewriter by IBM with a ball print head instead of jamming bars, which 
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could be easily replaced for different fonts and left the paper in place and moved the type ball 
instead 

1961 Information Flow in Large Communication Nets by Leonard Kleinrock 

1961 Synthesised Speech by John Larry Kelly, Jr and Louis Gerstman of Bell Labs 

1961 Expensive Typewriter by Steve Piner and L. Peter Deutsch 

1962 TECO (Text Editor & Corrector), both a character-oriented text editor/word processor 
and a programming language, by Dan Murphy 

1962 the Western Union Telegraph Company established its Telex system in the United States 
(where the name Telex is a registered trademark) 

1962 Highlighter Pen by Frank Honn 

1962 Modern fibre-tipped Pen by Yukio Horie at the Tokyo Stationery Company 

1962 Enciclopedia Mecanica by Angela Ruiz Robles 

1962 RUNOFF by Jerome H. Saltzer. Bob Morris and Doug Mcllroy (text editor with 
pagination) 

1962 The Structure of Scientific Revolutions by Thomas S. Kuhn 

1962 Spacewar! by Steve Russell in collaboration with Martin Graetz and Wayne Wiitanen 
1962 Augmenting Human Intellect: A Conceptual Framework by Doug Engelbart at SRI 
1963 Sketchpad (a.k.a. Robot Draftsman) software by Ivan Sutherland at MIT 

1963 The ‘smiley face’ by Harvey Ball, emoticon precursor 

1963 Augmentation Research Center by Doug Engelbart at SRI 


1963 Transport font, a sans serif typeface first designed for road signs in the United Kingdom 
by Jock Kinneir and Margaret Calvert 


1963 TJ-2 (Type Justifying Program) by Peter Samson (first page layout program) 
1963 ASCII (American Standard Code for Information Interchange) a character encoding 


standard for electronic communication developed from telegraph code 
1963 ‘Hypertext’ word coined by Ted Nelson 
1963 Computer Mouse and Chorded Keyset by Doug Engelbart 


1964 ELIZA natural language-like processing computer program by Joseph Weizenbaum at 
the MIT Artificial Intelligence Laboratory 


1964 LDX (Long Distance Xerography) by Xerox Corporation, considered to be the first 


commercial fax machine 
1964 Understanding Media by Marshall McLuhan 
1964 ASCII 7-bit standard 
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1964 TYPSET text formatting software used with the RUNOFF program 


1965 TV-Edit, one of the first CRT-based display editors/word processors that was widely 
used by Brian Tolliver for the DEC PDP-1 computer 


1965 Semi-Conductor based thermal printer by Jack Kilby at Texas Instruments 
1965 ‘Hypertext’ by Ted Nelson first in print, as well as first design (zipper lists) 
1965 MAIL Command for MIT’s CTSS, proposed by Pat Crisman, Glenda Schroeder and 


Louis Pouzin, implemented by Tom Van Vleck and Noel Morris 


1966 Object Oriented Programming by Ole-Johan Dahl and Kristen Nygard at the Norwegian 
Computing Center 

1966 Computers and the Humanities, Journal founded by Joseph Raben at Queens College in 
the City University of New York 

1967 HES (The Hypertext Editing System) co-designed at Brown University by Ted Nelson, 


Andy van Dam and Steve Carmody, as well as other student implementors, based in part on a 


spec Ted Nelson had written previously for Harcourt Brace 
1967 The Quick-Draw Graphics System masters thesis by Jef Raskin 
1967 Logo programming language designed by Wally Feurzeig, Seymour Papert, Cynthia 


Solomon at Bolt, Beranek and Newman 
1967 Newspapers use digital production processes and begin using computers for operations 


1968 A ‘low-tack’, reusable, pressure-sensitive adhesive accidentally created by Dr. Spencer 
Silver at 3M which would eventually be marketed as Post-it® Note 


1968 Doug Engelbart’s Seminal Demo of the NLS system at FJCC, including windows, 
hypertext, graphics, efficient navigation and command input, video conferencing, the 
computer mouse & chorded keyset, word processing, dynamic file linking and revision 
control 

1968 Dynabook Concept computer by Alan Kay 

1968 Digi Grotesk, digital typeface by Rudolph Hell 

1968 The Art of Computer Programming by Donald Knuth 

1968 OCR-A monospaced typeface for Optical Character Recognition by 23 American type 


foundries 


1968 OCR-B monospaced typeface by Adrian Frutiger for Monotype, following the 


European Computer Manufacturer’s Association standard 
1968 Serial Impact Dot Matrix Printer by OKI 


1968 SHRDLU natural language understanding computer program by Terry Winograd at 
MIT 
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1969 FRESS, inspired in part by HES and Engelbart’s NLS by Andy van Dam and his 


students at Brown University 


1969 GML, leading to SGML by Charles Goldfarb, Edward Mosher and Raymond Lorie at 
IBM 


1969 Ed line editor/word processor for the Unix, developed in by Ken Thompson 

1969 Vladamir Nabokov presents concept of emoticon/emoji to New York Times 

1969 Structured Writing and Information Mapping by Robert E. Horn 

1969 ARPANET based on concepts developed in parallel with work by Paul Baran, Donald 


Davies, Leonard Kleinrock and Lawrence Roberts 


1970 


1970s Gyricon Electronic Paper by Nick Sheridon at Xerox PARC 

1970 Xerox PARC founded by Jacob E. Goldman of Xerox 

1970 The Western Union Telegraph Company acquires TWX from AT&T 
1970 IBIS (issue-based information system) conceptualised by Horst Rittel 
1970 Journal by David A. Evans 


1970 Bomber by Len Deighton, first published novel written with the aid of a commercial 
word processor, the IBM’s MT/ST (IBM 72 IV) 


1970 Daisy Wheel Printing by Andrew Gabor at Diablo Data Systems allowing for 


proportional fonts 

1971 New York Times article refers to “the brave new world of Word Processing” 
1971 Laser Printer by Gary Starkweather at Xerox PARC 

1971 File Transfer Protocol (FTP) by Abhay Bhushan 

1971 Project Gutenberg by Michael S. Hart 

1971 Email with @ by Ray Tomlinson 


1971 PUB scriptable markup language. Brainchild of Les Earnest of the Stanford Artificial 
Intelligence Laboratory and implemented by Larry Tesler 


1972 TLG (Thesaurus Linguae Graecae) founded by Prof Marianne McDonald at the 
University of California, Irvine, to create a comprehensive digital collection of all surviving 


Greek texts from antiquity to the present era 
1972 C programming language by Dennis Ritchie and Ken Thompson 
1972 Xerox Star memo written by Butler Lampson, inspired by NLS 
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1973 Xerox Alto by Xerox PARC designed primarily by Charles P. Thacker 
1973 Addison-Wesley replaces its mechanical typesetting technology with computerised 


typesetting 
1973 Copy & Paste by Larry Tessler at Xerox PARC 


1973 Click & Drag by Jeff Raskin at Xerox PARC 


1973 Micral, first personal computer using a microprocessor by André Truong Trong Thi, 
Réalisation d'Etudes Electroniques (R2E), (Orsay, France) 

1973 Community Memory Bulletin Board precursor 

1974 Omni-Font Optical Character Recognition System (OCR) Scanners by Ray Kurzweil at 
Kurzweil Computer Products 

1974 Bravo word processor by Butler Lampson, Charles Simonyi at Xerox PARC. They 
would go on to produce Word 

1974 Computer Lib/Dream Machines by Ted Nelson 

1974 ‘Writing with light, writing on glass’ were the closing words of Wilfred A. Beeching’s 
Century of the Typewriter 

1974 Transmission Control Protocol (TCP) an internet working protocol for sharing resources 
using packet switching among network nodes forming the foundation of the Internet (short 


for internet working) 


1975 ZOG by Allen Newell, George G. Robertson, Donald McCracken and Robert Akscyn at 
Carnegie Mellon University 

1975 Microsoft founded by Bill Gates and Paul Allen 

1975 MUSA Speech Synthesis systems (MUltichannel Speaking Automaton) project led by 
Giulio Modena 

1975 Altair 8800 computer by Ed Roberts and Forrest M. Mims III 

1975 Gypsy document preparation system/word processor by Larry Tesler, Timothy Mott, 
Butler Lampson, Charles Simonyi, with advice from Dan Swinehart and other colleagues 
1975 Colossal Cave Adventure text adventure game by Will Crowther and later expanded by 
Don Woods 

1976 Second edition of The Art of Computer Programming by Donald Knuth, published by 
Addison-Wesley, which was typeset using phototypesetting which inspired him to develop 
TeX since he found the typesetting inferior to the original, Monotype typeset edition 

1976 Frutiger series of typefaces by Adrian Frutiger 

1976 Apple Computer (later Apple Inc.) founded Steve Jobs, Steve Wozniak and Ronald 
Wayne 
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1976 The Metanovel: Writing Stories by Computer by James Meehan 


1976 Emacs (Editor MACroS) word processor by David A. Moon, Guy L. Steele Jr. and 
Richard M. Stallman, based on TECO 


1976 vi word processor by Bill Joy (now Vim) 
1976 PROMIS (Problem-Oriented Medical Information System) by Jan Schultz and 


Lawrence Weed the University of Vermont 
1977 Apple II computer by Steve Wozniak at Apple 
1977 DataLand developed at MIT 


1977 Zork interactive fiction computer game by Tim Anderson, Marc Blank, Bruce Daniels, 
and Dave Lebling at MIT 


1977 Inkjet Printing by Ichiro Endo at Canon 
1977 Preliminary Description of TEX Memo by Donald Knuth 


1977 Name/Finger protocol (provided status on a particular computer system or person at 


network sites) by Harrenstien 


1978 Aspen Movie Map, the first hypermedia/interactive videodisc by Andy Lippman, Bob 
Mohl and Michael Naimark of the MIT Architecture Machine Group 


1977 Personal computers as dynamic multimedia by Alan Kay and Adele Goldberg 
1978 Public dial-up BBS by Ward Christensen and Randy Suess 


1978 TeX by Donald Knuth released as the first version which was used by others. Written in 
SAIL (Stanford Artificial Intelligence Language) 


1978 American Mathematical Society Gibbs Lecture by Donald Knuth, Mathematical 
Typography; published in the Bulletin (New Series) of the American Mathematical Society, 
volume 1, 1979, pp. 337-372 


1978 Vancouver Citation Style (author number), as a part of the Uniform Requirements for 
Manuscripts Submitted to Biomedical Journals (URMs) 


1978 QuarkXPress desktop publishing software by Quark 


1978 Earliest documented electronic Spam (although the term had not yet been coined) by 
Gary Thuerk 


1978 LISA computer by Apple design starts, with a requirement for proportional fonts 
1978 Speak & Spell by Texas Instruments 

1978 Highlighters with fluorescent colours by Dennison Company 

1978 Wordstar word processor by Rob Barnaby 

1979 WordPerfect word processor by Bruce Bastian and Alan Ashton at Brigham Young 


University 
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1979 Hayes Modem by Dennis C. Hayes and Dale Heatherington 

1979 Metafont by Donald Knuth 

1979 -) proposed by Kevin Mackenzie as a joke-marker precursor emoticon 
1979 Architext by Genette, Gerard. Hypertext as based on a hypotekst 

1979 EasyWriter for Apple II by John Draper 


1979 TV-EDIT word processor was used by Douglas Hofstadter to write ‘Gédel, Escher, 
Bach’ 


1979 Macintosh Project started by Jef Raskin and included Brian Howard, Marc LeBrun, 
Burrell Smith, Joanna Hoffman, and Bud Tribble. Named for Raskin’s favourite apple, the 
succulent McIntosh. He changed the spelling of the name to avoid potential conflict with the 


audio equipment manufacturer named McIntosh 
1979 Post-Its® by 3M sold commercially 
1979 Steve Jobs visited Xerox PARC, organized by Jef Raskin, as part of an investment 


agreement 


1980 


1980s SPAM used as a term to describe users on BBSs and MUDs who repeat it a huge 
number of times to scroll other users’ text off the screen. It later came to be used on Usenet to 


mean excessive multiple postings 
1980s Telex usage goes into decline as fax machines grow in popularity 
1980 ZX80 by Sinclair 


1980 Smalltalk designed by Alan Kay, Dan Ingalls, Adele Goldberg and developed by Alan 
Kay, Dan Ingalls, Adele Goldberg, Ted Kaehler, Diana Merry, Scott Wallace, Peter Deutsch at 
the Learning Research Group of Xerox PARC 


1980 PC by IBM 

1980 Imagen founded by Les Earnest, sold to QMS in 1987 
1980 Floppy Disks become prevalent for personal computers 
1980 Vydec1800 Series Word Processor by Exxon 

1980 ENQUIRE proposed by Tim Berners-Lee 

1980 USENET by Tom Truscott and Jim Ellis 


1982-3 The Encyclopaedia Project by Alan Kay, Charles Van Doren, Brenda Laurel, Steve 
Weyer and Bob Stein at Atari Research Group 


1981 Movie Manual by David Backer at the MIT Architecture Machine Group 
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1981 Raskin leaves the Macintosh project and Steve Jobs takes over 

1981 BITNET, EARN and NetNorth network university IBM mainframes, allowing text 
(mail, files, chat) to be shared by non-Arpanet institutions 

1981 TPS (Technical Publishing Software) by David Boucher at Interleaf, allowed authors to 
write text and create graphics WYSIWYG 

1981 First major use of Information Murals in Organizations by David Sibbet 

1982 Guide by Peter J. Brown at Canterbury University 

1982 Adobe founded by John Warnock and Charles Geschke 

1982 First ASCII emoticons :-) and :-( by Scott Fahlman at Carnegie Mellon University 
1982 CD-ROM by Denon 

1982 Tron movie released, the first movie written on a computer, an Alto at PARC. Written 
by Bonnie MacBird based on inspiration by Ted Nelson’s Computer Lib with consultation 
from Alan Kay, whom Bonnie would later marry 

1982 TeX82, a new version of TeX, rewritten from scratch, renaming the original TeX TeX78 
1983 Viewtron by AT&T and Knight Ridder 

1983 MILNET physically separated from ARPANET 

1983 ThinkTank outliner for Apple I 

1983 ARPANET switches to TCP/IP 

1983 Lisa by Ken Rothmuller, replaced by John Couch with contributions from Trip 
Hawkins, Jef Raskin and Steve Jobs, at Apple 

1983 Word word processor for DOS by Charles Simonyi and Richard Brodie for Xenix (Unix 
OS) and MS-DOS, at Microsoft. Originally called ‘Multi-Tool Word’ 

1983 KMS (Knowledge Management System), a descendant of ZOG by Don McCracken and 
Rob Akscyn at Knowledge Systems (a spinoff from the Computer Science Department of 
Carnegie Mellon University) 

1983 Hyperties by Ben Shneiderman at the University of Maryland 

1983 Multi-Tool Notepad word processor by Richard Brodie at Microsoft 

1983 ‘1984’ Macintosh Television Commercial by Apple 

1984 Literate Programming introduced by Donald Knuth, and approach to treat a program as 
literature understandable to human beings. Implemented at Stanford University as a part of 
research on algorithms and digital typography under the name WEB 


1984 Macintosh launched. In addition to the original contributors, the team also included Bill 
Atkinson Chris Espinosa, Joanna Hoffman, George Crow, Bruce Horn, Jerry Manock, Susan 
Kare, Andy Hertzfeld, and Daniel Kottke 
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1984 MacWrite word processor included with Macintosh, by Randy Wigginton, Don Breuner 
and Ed Ruder of Encore Systems for Apple. Also known as ‘Macintosh WP’ (Word 


Processor) and ‘MacAuthor’ before release 


1984 The Print Shop designed by David Balsam and programmed by Martin Kahn at 
Broderbund 


1984 Metafont by Donald Knuth updated to a version still in use at the time of writing this 
book 


1984 FidoNet bulletin board system software by Tom Jennings 

1984 LaserWriter printer by Apple 

1984 ‘Cyberspace’ term coined by William Gibson in Neuromancer 

1984 Organizer by David Potter at Psion 

1984 PostScript by John Warnock, Charles Geschke, Doug Brotz, Ed Taft and Bill Paxton at 
Adobe, influenced by Interpress, developed at Xerox PARC 

1984 MacroMind founded by Marc Canter, Jay Fenton and Mark Stephen Pierce 

1984 PC Jr desktop computer by IBM 

1984 Notecards by Randall Trigg, Frank Halasz and Thomas Moran at Xerox PARC 


1984 Highlighted Selectable Link by Ben Shneiderman and Dan Ostroff at University of 
Maryland 


1984 TIES by Ben Shneiderman at University of Maryland 
1984 LaserJet by HP 


1984 Text Messaging / SMS (short message service) developed by Franco-German GSM 
cooperation by Friedhelm Hillebrand and Bernard Ghillebaert 


1984 Filevision by Telos 

1984 LaTeX by Leslie Lamport who was writing a book and needed macros for TeX, 
resulting in ‘Lamport’s TeX’ (‘LaTeX’) 

1984 Zoomracks for Atari by Paul Heckel 

1985 Symbolics Document Examiner by Janet Walker 

1985 Guide, commercial edition, by OWL (Office Workstations Ltd) 

1985 Pagemaker desktop publishing software by Aldus, bought by Adobe in 1994 
1985 StarWriter word processor by Marco Borries at Star Division 

1985 Intermedia by Norman Meyrowitz and others at Brown University 

1985 Windows operating system spearheaded by Bill Gates at Microsoft 

1985 Write word processor by Microsoft, included with Windows 
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1985 Word word processor by Microsoft ported to Macintosh 

1985 Amiga computer by Commodore 

1985 Emacs General Public License by Richard Stallman, the first copyleft license 

1985 TRICKLE by Turgut Kalfaoglu at Ege University, Izmir; BITNET-to-Internet gateway 
allows sharing of text and programs between two disparate networks 

1986 Guide by Peter J. Brown at the University of Kent, marketed by OWL 

1986 Harvard Graphics desktop business application by Software Publishing Corporation 
1986 Texinfo GNU Documentation System by Richard Stallman and Bob Chassell, 
developed by Brian Fox and Karl Berry 


1986 FrameMaker document/word processor by Frame Technology. Developed by Charles 
‘Nick’ Corfield based on an idea from Ben Meiry and commercialised with Steve Kirsch. 
Bought by Adobe 1995 

1986 Hyperties commercial version by Cognetics Corporation 

1986 Solid Ink Printing by Tektronix 

1986 SGML (Standard Generalized Markup Language), ISO 8879 

1986 Uncle Roger by Judy Malloy released on Art Com Electronic Network on The Well 
1987 PowerPoint presentation software created by Robert Gaskins and Dennis Austin at 
Forethought Inc., bought by Microsoft same year and released as a Microsoft product 1989 
1987 MacroMind Director multimedia authoring by MacroMind 

1987 V.I.P. (Visual Interactive Programming) by Dominique Lienart at Mainstay Inc 

1987 Storyspace by Jay David Bolter & Michael Joyce, maintained and distributed by Mark 
Bernstein of Eastgate Systems 

1987 Afternoon a story, by Michael Joyce, first digital hypertext narrative 

1987 Unicode by Joe Becker from Xerox with Lee Collins and Mark Davis from Apple 
1987 Franklin Spelling Ace by Franklin Electronic Publishers 

1987 Canon Cat by Jef Raskin at Canon Inc 

1987 Apple Knowledge Navigator visionary concept video initiated by John Sculley, 
sponsored by Bud Colligan, written and creatively developed by Hugh Dubberly and Doris 
Mitsch with input from Mike Liebhold and advice from Alan Kay, inspired by the MIT Media 
Lab, with product design by Gavin Ivester and Adam Grosser at Apple 

1987 TEI (Text Encoding Initiative) ‘Poughkeepsie Principles’: text encoding guidelines for 
Humanities texts 

1987 HyperCard by Bill Atkinson at Apple 

1987 Amanda Goodenough’s children’s point and click stories in Hypercard published by 
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Voyager 

1987 Hypertext’87 First ACM conference on hypertext 

1988 Microcosm by Wendy Hall, Andrew Fountain, Hugh Davis and Ian Heath 

1988 NeXT Cube by NeXT 

1988 IRC by Jarkko Oikarinen 

1988 Think’n Time (Visual outliner with dates) by Benoit Schillings & Alain Marsily at 
Mainstay Inc 

1988 # (hash) and & (ampersand) used in IRC to label groups and topics (RFC 1459) 
1988 Wolfram Mathematica by Stephen Wolfram 

1988 Hypertext edition of Communications of the ACM using Hyperties by Ben 
Shneiderman 

1988 Idex by William Nisen of Owl, based on Guide 

1988 Hypertext Hands-On! by Ben Shneiderman and Greg Kearsley, first commercial 


electronic book 


1988 Reflections on NoteCards: seven issues for the next generation of hypermedia systems 
by Frank,G. Halasz 


1988 Serial Line Internet Protocol (SLIP) by J. Romkey 
1988 Breadcrumb Trail navigation metaphor in Hypergate by Mark Bernstein 
1989 GRiDPad 1900, the first commercial tablet by GRiD Systems Corporation 


1989 Robert Winter’s CD Companion to Beethoven’s Ninth Symphony, published by 
Voyager, the first viable commercial CD-ROM 


1989 Markup (Visual document annotations with markup signs - Groupware) by Dominique 


Lienart & all at Mainstay Inc 
1989 SuperCard by Bill Appleton at Silicon Beach Software 


1989 gIBIS by Jeff Conklin and Michael Begeman, commercialised in the 1990s as CM/1 
and QuestMap 


1989 Bidirectional Email-to-Fax Gateway hosted by UCC 
1989 Word for Windows word processor by Microsoft 
1989 Mapping Hypertext: Analysis, Linkage, and Display of Knowledge for the Next 


Generation of On-Line Text and 
Graphics by Robert E. Horn 


1989 Information Management: A Proposal by Tim Berners-Lee at CERN. World Wide Web 
protocols published on USENET in alt.hypertext 
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1990 


1990s T9 invented by Martin King and Cliff Kushler, co-founders of Tegic 

1990s Compendium by Al Selvin and Maarten Sierhuis 

1990 MarcoPolo (Visual Document Management - Groupware) by Benoit Schillings & Alain 
Marsily at Mainstay Inc 


1990 Archie, a tool for indexing FTP archives, considered to be the first Internet search 
engine, by Alan Emtage and Bill Heelan at McGill University/Concordia University in 


Montreal 

1990 Python programming language by Guido van Rossum 

1990 The SGML Handbook by Charles F. Goldfarb 

1990 Designing Hypermedia for Learning by David H. Jonassen and Heinz Mand! (editors) 


in which updated conference proceedings are annotated by the authors with typed hypertext 


links in the margins connecting passages between the articles 


1991 Gopher protocol by the University of Minnesota (initial version of the protocol 
appeared in 1991, codified in 1993 as a RFC 1436) 

1991 Seven Issues: Revisited Hypertext ‘91 Closing Plenary by Frank G. Halasz at Xerox 
Corporation 

1991 World Wide Web by Tim Berners-Lee becomes the first global hypertext system 

1991 DocBook DTD by HaL Computer Systems and O’Reilly & Associates 

1991 Camelot Project started as in at Adobe, later to become PDF 

1991 PowerBook Laptops by Apple 

1991 Aquanet by Catherine C. Marshall, Frank G. Halasz, Russell A. Rogers and William C. 


Janssen Jr. 

1991 Visual Basic programming language by Microsoft 

1991 Java programming language project launched by James Gosling, Mike Sheridan and 
Patrick Naughton. Originally called Oak, then Green, and finally Java 

1991 Instant Update by ON Technology 


1991 HTML by Tim Berners-Lee, influenced by SGML guid, an in-house markup language at 
CERN 


1991 CURIA (now CELT: Corpus of Electronic Texts) first corpus in Early Irish to be 
published on the World-Wide Web by University College Cork, Ireland 


1991 Expanded Books Project by The Voyager Company 
1991 TeachText by Apple, included with System 7 


593 


1992 First Text Message (SMS) is sent by Neil Papworth reading: “Merry Christmas” to 


Richard Jarvis at Vodafone 


1992 Veronica a search engine system for the Gopher protocol by Steven Foster and Fred 


Barrie at the University of Nevada, Reno 


1992 Lynx internet web browser by Lou Montulli, Michael Grobe, and Charles Rezac at the 


University of Kansas 
1992 Frontier by Dave Winer at UserLand Software released on Mac 


1992 OpenDoc by Kurt Piersol and Jed Harris at Apple. First code named ‘Exemplar’, then 
‘Jedi’ and ‘Amber’ 


1992 Palm Computing founded by Jeff Hawkins 
1992 The End of Books By Robert Coover, Hypertext fiction cover story in the New York 


Times Book Review 

1992 Before Writing by Denise Schmandt-Besserat 

1992 Portable Document Format (PDF) by Adobe 

1992 BBEdit word processing software by Rich Siegel at Bare Bones Software 
1993 Mosaic web browser by Marc Andreessen and Eric Bina at NCSA massively 


popularises the web 


1993 Microsoft Word word processor celebrates its 10th anniversary with 10 million Word 


users 
1993 Encarta multimedia encyclopedia by Microsoft 
1993 Hypermedia Encyclopedias sell more copies than print encyclopedias 


1993 Newton MessagePad PDA by Steve Sakoman, Steve Capps, Larry Tesler, Michael 
Culbert, Michael Tchao and others at Apple under John Sculley 


1993 Early Blog by Rob Palmer 
1993 Open Agent Architecture (OAA) delegated agent framework by Adam Cheyer et al. at 


SRI International 
1993 Georgia typeface designed by Matthew Carter and hinted by Tom Rickner for Microsoft 


1993 Searching for the Missing Link: Discovering Implicit Structure in Spatial Hypertext by 
Catherine C. Marshall and Frank Shipman. First occurrence of Spatial Hypertext in print 


1993 AppleScript launched with System 7 by Apple 

1994 PDF made freely available 

1994 Links.net blog by Justin Hall, before the term would be used 
1994 TrueType Open by Microsoft 


1994 Point-to-Point Protocol (PPP) enabled internet communications between two routers 
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directly by W. Simpson 

1994 Netscape Navigator web browser by Jim Clark and Marc Andreessen at Netscape 
Communications Corp 

1994 Scripting News by Dave Winer 

1994 Yahoo! founded by Jerry Yang and David Filo 

1994 Amazon founded by Jeff Bezos 


1994 Semantic Web vision presented by Tim Berners-Lee at the first World Wide Web 
Conference 


1994 QR Code System by the Japanese company Denso Wave, a subsidiary of Toyota 
1994 World Wide Web Consortium founded 


1994 PageMill HTML authoring by Seneca Inc., bought by Adobe one year later, 
discontinued 2000 


1994 VIKI: Spatial Hypertext Supporting Emergent Structure by Catherine C. Marshall, 
Frank M. Shipman HI, James H. Coombs 


1994 A Subversive Proposal by Stevan Harnad at the University of Southampton 
1995 WordPad word processor by Microsoft is included in Windows 95, replacing Write 


1995 Netscape goes public and gains market value of almost $3B on first day of stock market 
trading 


1995 The World Wide Web Handbook by Peter Flynn, first comprehensive book on HTML 
1995 Ruby scripting langauge by Yukihiro ‘Matz’ Matsumoto 

1995 Windows 95 operating system by Microsoft 

1995 WikiWikiWeb, the first wiki, by Ward Cunningham 


1995 Java public release by James Gosling at Sun Microsystems (since been acquired by 


Oracle), the first programming language to use Unicode for all text 


1995 JavaScript by Brendan Eich at Netscape (orignally called Mocha, then LiveScript and 
later JavaScript) 


1995 AltaVista founded by Paul Flaherty, Louis Monier, Michael Burrows and Jeffrey Black 
1995 FutureSplash by FutureWave, sold to Macromedia in 1996 and renamed Flash 


1996 Cascading Style Sheets (CSS) by Hakon Wium Lie and Bert Bos at the World Wide 
Web Consortium 


1996 Palm OS PDAs including the Graffiti handwriting system 
1996 Vaio laptop by Sony 
1996 Cyberdog OpenDoc based Internet suite of applications by Apple 
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1996 OpenType by Microsoft joined by Adobe 

1996 Anoto by Christer Fahreeus to provide digital pen capability to paper 

1996 Hotmail email system by Sabeer Bhatia and Jack Smith, bought by Microsoft in 1997 
1996 The Internet Archive by Brewster Kahle 


1996 GoLive HTML authoring software by GoNet Communication, Inc., bought by Adobe 
1999 

1996 TextEdit word processor by Apple. Not meant for use, it was sample code 

1996 Live word count by Keith Martin, demonstrated in the Wordless word processor, later 
appearing in Microsoft Word 98 

1997 Emoji developed by Japanese mobile operators during the 1990s including SoftBank 
and Shigetaka Kurita for i-mode 

1997 Meta Content Framework developed by Ramanathan V. Guha at Apple Computer’s 
Advanced Technology Group, leading to RDF 

1997 OpenDoc by Apple cancelled 

1997 Apple Data Detectors by Jim Miller, Thomas Bonura and others at Apple’s Advanced 
Technology Group, which would also lead on to LiveDoc 

1997 Resource Description Framework (RDF) derived from W3C’s PICS, Dublin Core and 
from the Meta Content Framework (MCF) developed by Ramanathan V. Guha at Apple and 
Tim Bray at Netscape 

1997 Dreamweaver HTML authoring software by Macromedia, bought by Adobe 2005 
1997 Yandex by Arkady Volozh and Ilya Segalovich 

1997 Flash multimedia authoring and platform by Macromedia, later bought by Adobe 
1997 ‘weblog’ term coined by Jorn Barger to describe a log of his internet activity 

1997 Jabberwacky released online by Rollo Carpenter 

1997 E-Paper by Barrett Comiskey, Joseph Jacobson and JD Albert at E Ink Corporation 
1997 Newton PDA by Apple cancelled after Steve Jobs return 

1997 Unistroke by David Goldberg at Xerox PARC 

1997 90001 Communicator monile phone by Nokia, the first mobile phone with a full 
keyboard 

1997 OpenType by Microsoft 

1997 Liquid Mail email system by Frode Alexander Hegland featuring smart Views 

1998 iMac desktop computer by Apple 

1998 First blog published on an established news site by Jonathan Dube at The Charlotte 


Observer 
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1998 Can Computers Think? History and Status of the Debate. Seven posters. Industrial 


strength argumentation map by Robert E. Horn 
1998 Open Diary blogging service by Bruce Ableson 
1998 Visual Language: Global Communication for the 21st Century Robert by E. Horn 


1998 (possibly 1999) Fluid Links demo video at the ACM CHI conference by Polle T. 
Zellweger, Bay-Wei Chang, and Jock D. Mackinlay 


1998 ‘SPAM’ in The New Oxford Dictionary of English 
1998 Google founded by Larry Page and Sergey Brin 
1998 XML 1.0 becomes a W3C Recommendation 

1998 Netscape goes open source with the name Mozilla 


1998 XML-RPC text-based networking protocol between apps running across operating 


systems 

1998 Frontier blog software by Dave Winer at UserLand Software released on Windows 
1998 MathML by W3C 

1998 @font-face by W3C 

1998 AOL buys Netscape for $4 Billion 

1999 Open eBook 

1999 The short form, ‘blog’, was coined by Peter Merholz. Shortly thereafter, Evan Williams 


at Pyra Labs used ‘blog’ as both a noun and verb and devised the term ‘blogger’ in 


connection with Pyra Labs' Blogger product, leading to the popularization of the terms 
1999 LiveJournal blogging service by Brad Fitzpatrick at Danga Interactive 


1999 Blogger blogging service by Evan Williams and Meg Hourihan with significant coding 
by Paul Bausch and Matthew Haughey 


1999 RDF Site Summary (RSS 0.9) the first version of RSS, by Dan Libby and Ramanathan 
V. Guha at Netscape 


1999 RSS 0.91 by Dave Winer at UserLand 

1999 my.netscape.com and my.userland.com 

1999 Edit This Page by Dave Winer 

1999 Code and Other Laws of Cyberspace by Larry Lessig 

1999 Mac OS X operating system by Apple 

1999 Ajax web development techniques for asynchronous web applications emerges 


1999 ActiveText: A Method for Creating Dynamic and Interactive Texts by Jason E. Lewis 


and Alex Weyers at Interval Research Corporation 
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1999 Spatial Hypertext: An Alternative to Navigational and Semantic Links by Frank M. 
Shipman and Catherine C. Marshall 


1999 Electronic Literature Organization (ELO) founded by Scott Rettberg, Robert Coover, 
and Jeff Ballowe 


2000 


2000 Optical Character Recognition (OCR) software is made available online for free 
2000 1 billion indexable pages on the Web, estimated by NEC-RI and Inktomi 

2000 ClearType by Microsoft 

2000 XML Linking Language (XLink) an XML markup language for creating internal and 


external links within XML documents, and associating metadata with those links, by Steven 
DeRose, Eve Maler, David Orchard and Bernard Trafford 


2000 EPrints by Stevan Harnad, funded by Wendy Hall, supervised by Les Carr and 
implemented by Rob Tansley and others at the University of Southampton 

2000 CoolType by Adobe 

2000 ScholOnto by Simon Buckingham Shum, Enrico Motta and John Domingue at the 
Knowledge Media Institute, The Open University. This evolved over the next decade into 
ClaiMaker and Cohere with Victoria Uren, Gangmin Li, Anna De Liddo and Michelle 
Bachler 


2000 Riding the Bullet by Stephen King, the first mass-market e-book for encrypted 


download 

2000 EverNote founded by Stepan Pachikov 

2001 ‘Chinese General Language and Character Law’ rolled out. 
2001 Tinderbox by Mark Bernstein, Eastgate Systems 


2001 Semantic Web vision popularised in a Scientific American article by Tim Berners-Lee, 
James Hendler and Ora Lassila 

2001 G4 Titantum PowerBook laptop computer by Apple 

2001 The Wiki Way by Bo Leuf and Ward Cunningham 

2001 Creative Commons by Lawrence Lessig, Hal Abelson, and Eric Eldred 

2001 Wikipedia online collaborative encyclopedia by Jimmy Wales and Larry Sanger at 
Nupedia 

2001 Movable Type weblog publishing system by Benjamin Trott and Mena Grabowski Trott 
at Six Apart 
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2001 JSON by Douglas Crockford 
2001 Douglas Adams’ speech about Virtual Graffiti held at the 3GSM World Congress 
2002 Bibliotheca Alexandrina founded, the modern Library of Alexandria, with Ismail 


Serageldin as the founding director 

2002 EPrints version 2 lead developer Christopher Gutteridge 

2003 Android Inc founded by Andy Rubin, Rich Miner, Nick Sears, and Chris White 
2003 Friendster social media service Jonathan Abrams 


2003 Myspace blogging and social media service by Brad Greenspan, Josh Berman and Tom 


Anderson at eUniverse 
2003 Deep Love by Yoshi, first cell phone novel ( Japanese ‘Keitai Shousetsu’) 
2003 The Legal Deposit Libraries Act widens the definition of what publishers should send to 


the libraries to include digital publications, pending further regulation 

2003 WordPress blogging service by Matt Mullenweg and Mike Little 

2003 Blogger blogging service is bought by Google 

2003 TypePad blogging service by BizLand, later Endurance International Group (EIG) 
2003 Ulysses word processor by Max Seelemann and Marcus Fehn 


2004 Facebook social media service by Mark Zuckerberg, Eduardo Saverin, Andrew 
McCollum, Dustin Moskovitz and Chris Hughes 


2004 First hypertext format full length articles accepted at ACM’s Hypertext Conference with 


Twin media: hypertext structure under pressure by David Kolb awarded ‘Best Paper’ 


2004 First hypertext format article at ACM’s Document Engineering conference by James 


Blustein and Mona Noor 

2004 Institute for the Future of the Book founded by Bob Stein 
2004 Tag Cloud at Flickr, Technorati, WordPress Plugins and more 
2004 Scala programming language by Martin Odersky 

2005 Pages word processor by Apple 

2005 Markdown by John Gruber collaboration with Aaron Swartz 
2006 Time Person of the Year is ‘You’ 


2005 Writely by programmers Sam Schillace, Steve Newman and Claudia Carpenter at 
Upstartle 


2006 Upstartle bought by Google 
2006 Google Docs by Google 


2006 Twitter social media service founded by Jack Dorsey, Noah Glass, Biz Stone and Evan 
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Williams at Twitter 
2006 One Laptop Per Child by Nicholas Negroponte 


2006 HyperScope Project by Doug Engelbart and Brad Neuberg, Eugene Kim, Jonathan 
Cheyer and Christina Engelbart 


2006 Hyperwords Project by Frode Hegland, Fleur Klijnsma and Rob Smith 
2006 Office Open XML by Microsoft 


2006 The Semantic Web Revisited by Tim Berners-Lee, Nigel Shadbolt, and Wendy Hall, in 
IEEE Intelligent Systems 


2006 Debategraph by Peter Baldwin and David Price 


2006 Gamer Theory by McKenzie Wark’s, the first networked book, produced by the Institute 
for the Future of the Book 


2006 Dialogue Mapping: Creating Shared Understanding of Wicked Problems by Jeff 
Conklin 


2007 Hashtag by Chris Messina (name by Stowe Boyd) 

2007 iPhone by Apple Inc. 

2007 Kindle by Amazon 

2007 Scrivener for macOS by Keith Blount at Literature & Latte 
2007 EPUB by IDPF 

2008 MacBook Air by Apple 

2008 Last Stable Build of Netscape Navigator 

2009 Like Button by Facebook 

2009 Webfonts by Typekit 

2009 OmmWriter by Herraiz Soto & Co 

2009 iPhone Copy & Paste by Apple 

2009 Twine open-source tool for authoring interactive fiction by created by Chris Klimas 


2009 Worst year in decades as far as advertising revenues for newspapers and newspapers 
begin moving online 

2010 
2010 Thumbs Up Emoji 


2010 Retina Display by Apple 
2010 iA Writer word processor by Oliver Reichenstein 
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2010 iPad tablet by Apple 

2010 Swift programming language development by Chris Lattner, with the eventual 
collaboration of many other programmers at Apple 

2010 Siri developed by Dag Kittlaus, Tom Gruber, and Adam Cheyer, bought by Apple 
2010 Emoji ratified as part of Unicode 6.0 

2011 iMessage by Apple 

2011 ByWord word processor by Metaclassy 

2011 Scrivener word processor for Windows by Keith Blount at Literature & Latte 
2011 Annual Future Of Text Symposium by Frode Alexander Hegland launched 

2011 Liquid text utility by Frode Alexander Hegland at The Liquid Information Company 
2011 Siri personal digital assistant released as part of the iPhone 4S by Apple 

2011 Swype by Cliff Kushler allying users to drag their fingers on a virtual keyboard to 


connect the dots between letters 


2011 ClaiMaker by Gangmin Li, Victoria Uren, Enrico Motta, Simon Buckingham Shum and 


John Domingue 

2012 Knowledge Graph by Emily Moxley, Google’s lead product manager, at Google 
2012 Muse by Adobe 

2012 The Web-Extended Mind by Paul Smart 

2012 Inventing on Principle presentation by Bret Victor 

2012 Google Now Assistant launched by Google 

2012 Medium online social publishing platform by Evan Williams 

2012 LiquidText by Craig Tashman 

2012 Outlook by Microsoft replaces Hotmail 

2013 Non-Print Legal Deposit Regulations further define the digital elements of the Legal 


Deposit Libraries Act and lead to large-scale on-going transfer of e-journals and e-books to 


the legal deposit libraries for posterity 
2013 Distant Reading by Franco Moretti 


2013 First Full-Scale Harvest of the UK Domain by the UK Web Archive, using the Non- 
Print Legal Deposit Regulations 


2013 Ulysses III (major rewrite) by Max Seelemann and Marcus Fehn 
2014 Xanadu by Ted Nelson 
2014 Alexa assistant released by Amazon 


2014 Cortana assistant released by Microsoft 
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2014 Framtidsbiblioteket (The Future Library project) launched, a public artwork that aims to 


collect an original work by a popular writer every year from 2014 to 2114 


2014 Author reboot by Frode Hegland at The Liquid Information Company with coding by 


Jacob Hazelgrove 

2014 Most up to date version of TeX is 3.14159265 as of the publication of this book 

2014 Swift programming language launched at the Apple Worldwide Developers Conference 
(WWDC) 

2014 Author iOS by Frode Hegland at The Liquid Information Company 

2014 Augmented Writing by Textio 

2015 Notion by Ivan Zhao at Notion Labs 

2015 Watch by Apple 

2015 Hamilton musical, by Lin-Manuel Miranda, makes it Broadway debut, highlighting the 


beauty and power of the written word, with an opening line stating that Hamilton “put a 


pencil to his temple, connected it to his brain” 

2016 Reactions, also-called Tapback, for iMessage by Apple 

2016 Universal Clipboard by Apple 

2016 Viv Labs, developed by Dag Kittlaus, Adam Cheyer and Chris Brigham, acquired by 


Samsung 

2016 Notion founded by Ivan Zhao and Simon Last 

2017 Roam Research founded by Conor White-Sullivan 

2017 Web Annotations Standardised by the W3C Web Annotation Working Group 
2018 Bixby Marketplace, an open assistant ecosystem based on Viv Labs Technology, 


launched by Samsung 


2019 Reader PDF viewer with Visual-Meta support by Frode Alexander Hegland at The 


Liquid Information Company with coding by Jacob Hazelgrove 


2020 


2020 Muse by Adobe discontinued 

2020 Flash by Adobe discontinued 

2020 iPad Keyboard with Trackpad by Apple 

2020 Adobe Liquid Mode for Easier PDF Viewing on Mobile Devices powered by Sensei 


Machine Learning 
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Future 


2023 (Jan 1), Adobe Type 1 (Postscript) fonts reach end of life; no further support in Adobe 


products (other software unaffected) 


unknown The “absolutely final change (to be made after my death)” of TeX will be to change 


the version number to 7, 


at which point all remaining bugs will become features. Likewise, versions of Metafont after 
2.0 asymptotically approach e (currently at 2.7182818), and a similar change will be applied 
after Knuth’s death. 


unknown All the pioneers of digital text will die, leaving it to future generations to rediscover 


and hopefully improve upon how we interact with our textual knowledge, and each other. 


unknown You will read this. What will you do with what you have learnt in this book, what 


will you think of the way we 


saw text in 2020, how do you think the way we present and interact with text can be 


improved? 


Contributors to the Timeline 


Frode Hegland and Mark Anderson editors, with Peter Flynn, Mark Bernstein, Bernard 


Vatant, Bob Horn, Jonathan Finn, Niels Ole Finnemann and more. Thank you. 
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Colophon 


Notes on style: 

¢ In talks and discussions, the speaker’s name is notes before their spoken passage marked 
thus: ‘ speaker name: ... ’. 

* Otherwise any other ad hoc bolding in the body text of articles should be treated as an 
editorial highlight. 

¢ Insome places URLs are deliberately placed on a separate line to ensure PDF processing 
don’t break URL function. 


Published 2022. All articles are © Copyright of their respective authors. This collected work 
is © Copyright ‘Future Text Publishing’ and Frode Alexander Hegland. A PDF is made 
available at no cost and the printed book is available from ‘Future Text 

Publishing’ (futuretextpublishing.com) a trading name of ‘The Augmented Text Company 
LTD, UK. This work is freely available digitally, permitting any users to read, download, 
copy, distribute, print, search, or link to the full texts of these articles, crawl them for 
indexing, pass them as data to software, or use them for any other lawful purpose, without 
financial, legal, or technical barriers other than those inseparable from gaining access to the 
internet itself. The only constraint on reproduction and distribution, and the only role for 
copyright in this domain, should be to give authors control over the integrity of their work 


and the right to be properly acknowledged and cited. 
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Gallery from the Symposium 


The 11th Annual Future of Text Symposium was held at the Linnean Society in London on 
the 27th and 28th of September 2022 and online. Below are a few photographs from the 
event. These are not intended to be a complete record but rather for experimenting in XR 


with how attached images can be displayed. 
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Glossary 


liquid the document format for the Author Software. It is a macOS wrapper with JOSN and 
RTFD. This is a free and open standard optimised for VR. 


#Infrastructure Timeline: - 


6DoF Six degrees of freedom (6DOF) refers to the freedom of movement of a rigid body in 
three-dimensional space. Specifically, the body is free to change position as forward/ 
backward (surge), up/down (heave), left/right (sway) translation in three perpendicular axes, 
combined with changes in orientation through rotation about three perpendicular axes, often 
termed yaw (normal axis), pitch (transverse axis), and roll (longitudinal axis). Three degrees 
of freedom (3DOF), a term often used in the context of virtual reality, refers to tracking of 


rotational motion only: pitch, yaw, and roll. Used by us in VR context. 


See: https://en.wikipedia.org/wiki/Six_degrees_of_ freedom 


academic someone who reads academic documents, usually associated with an academic 
institution, but not necessarily. 


#Profession | Timeline: 


ACM From Wikipedia: “The Association for Computing Machinery (ACM) is a US-based 
international learned society for computing. It was founded in 1947 and is the world's largest 
scientific and educational computing society.” 
https://en.wikipedia.org/wiki/Association_for Computing Machinery 

ACM’s ‘Hypertext’ Conference piloted Visual-Meta in 2021. 


#Institution. #Timeline: 1947-. USA 


Adam Wern Working on spatial hypertext software. 


Adam, Adam Wern Future Text Lab collborator. 


Addressability refers to how something can refer to something else. In the digital world this 
primarily means in a local file structure and a server based network. In academia is can refer 
to citation or references. In the physical world it can refer to a formal location layout, such as 
a flat so and so in house so and so, as well as co-ordinates. It can also man relative 


addressing, such as saying take a left after the third yellow house. 


#dougconcept #term 


Adobe “originally called Adobe Systems Incorporated, is an American multinational 
computer software company incorporated in Delaware[3] and headquartered in San Jose, 
California. It has historically specialized in software for the creation and publication of a 
wide range of content, including graphics, photography, illustration, animation, multimedia/ 
video, motion pictures, and print.” https://en. wikipedia.org/wiki/Adobe_Inc. 


Founded by: John Warnock and Charles Geschke. 


#institution. #timeline: December 1982. 


AI ‘Artificial intelligence’ is “intelligence demonstrated by machines, as opposed to the 
natural intelligence displayed by animals including humans.” https://en.wikipedia.org/wiki/ 


Artificial_intelligence 


#Technology. Timeline: 20th century-. Keywords: ML. Machine Learning. 


Alan Laidlaw https://twitter.com/alanlaidlaw 


#person. Timeline: 


Alan Laidlaw https://twitter.com/alanlaidlaw 


anaphora “the use of a word referring back to a word used earlier in a text or conversation, 
to avoid repetition, for example the pronouns he, she, it, and they and the verb do in I like it 


and so do they.” 


From Oxford Languges. 


Andreea Ion Cojocaru Andreea Ion Cojocaru is a licensed architect and a software 
developer. 

She is the co-founder and CEO of NUMENA, an award-winning German company. Andreea 
works at the intersection of traditional architecture and immersive technologies to develop 
projects that require a new approach to cognitive and spatial challenges. Currently, she is 
working on a tool that allows users to create, share and edit environments at 1:1 scale. She is 
also leading projects that mix physical and virtual elements to explore social and economic 
models based on collective agency across multiple spatial modalities. 

She is featured in issue 1.4 of our Journal: https://futuretextpublishing.com/journal/ 

#person. Timeline: -. Company: NUMENA. Lives in: Germany. Key interests VR / AR / AI, 
embodiment, cognitive neuroscience / cognition, architecture. Works with: Bob Horn, Claus 


Atzenbeck. 


Andy Campbell is the Digital Director for the arts/media organisation One to One 
Development Trust and the founder/director of Dreaming Methods - an immersive digital 
storytelling studio. 

A judge and speaker at the New Media Writing Prize since its inception in 2010, he has been 
producing electronic literature, digital art and experimental narrative games for over 25 years 


and has won many international awards. He is a speaker and workshop leader giving 


masterclasses in Unity and digitally delivered storytelling. Often called upon by international 
arts and culture organisations as well as universities across the globe, his work includes Lead 
Developer for the multi-award winning episodic digital novel Inanimate Alice and services to 
the Orient Foundation for Arts and Culture creating the world’s largest online archive of 
digitized Tibetan cultural resources. His most recent collaborative work Monoliths with Pilot 
Theatre and One to One Development Trust has been nominated for the Immersive Art/KR 
award at the 2022 BFI London Film Festival. 

https://twitter.com/dreamingmethods facebook.com/dreamingmethods https:// 
www.instagram.com/dreamingmethods/ 

#person. Timeline: 23 April 1975-. Born in: Halifax, UK. Company: One to One 
Development Trust and Dreaming Methods www.onetoonedevelopment.org 
www.dreamingmethods.com. Currently: Wakefield, UK. Interests: Digital fiction, digital art, 
VR, WebGL, HTMLS, video games, game engines, 3D graphics, software, programming, 
digital environments, immersive audio, AI, XR, TV, electronic music, film, cinema, books, 
experimental literature, poetry, literacy, engagement, accessibility, inclusivity, collaboration, 


education, web design, CSS 


Anne-Laure Le Cunff Founder of Ness Labs and contributor to The Future of Text volume 


1. 


Annie Murphy Paul From an interview in GQ: “We make better use of our cognitive 
resources, says Paul, when we use them in conjunction with “extra-neural” resources: our 
body (embodied cognition), our environment (situated cognition), and the people around us 
(distributed cognition). The brain evolved to move the body, to navigate through space, to 
interact with other people,” says Paul. 

“Those are these human strengths that we're totally putting aside when we focus on the brain 
and we think, “To get real thinking and real work done, I have to sit still, not talk to anybody, 


and just push my brain harder and harder.’ It's just not working very well.” 


https://www.gq.com/story/extended-mind-annie-murphy-paul 


Author of The Extended Mind. 


#person. Timeline: -. 


Apple HMD an as yet not released VR/AR headset which is expected to be announced late 
2022, early 2023 and which will enlarge the market considerably. 
#Hardware | Timeline: 2023 | Company: Apple 


Apurva Chitnis Head Of Engineering, koodos labs. 


#person 


AR “Augmented reality (AR) is an interactive experience of a real-world environment where 
the objects that reside in the real world are enhanced by computer-generated perceptual 
information, sometimes across multiple sensory modalities, including visual, auditory, haptic, 


somatosensory and olfactory.” 


https://en.wikipedia.org/wiki/Augmented_ reality 


AR includes the real reality, as opposed to VR which does not. 


AR includes the real reality, as opposed to VR which does not. 


“Augmented reality (AR) is an interactive experience of a real-world environment where the 
objects that reside in the real world are enhanced by computer-generated perceptual 
information, sometimes across multiple sensory modalities, including visual, auditory, haptic, 


somatosensory and olfactory.” 
https://en.wikipedia.org/wiki/Augmented_ reality 


#Technology | Timeline: 20th century- 


ARC Engelbart Concept: Augmentation Research Center, The name of Doug's lab at SRI 


where he proposed a system called H-LAM/T in 1962 and developed and in 1968 
demonstrated NLS: oNLine System, his platform for shared knowledge work research, later 
renamed Augment and from which I decided on the name Author, since Author and Augment 


share etymological roots. 


#Institution. Timeline: 1960s-1970s. USA 


Architectural Spaces VR Represenations of landscapes, buildings, and interiors designs 


modeled after the physical world. 


Art a category used here to help the reader identify artist and discussions around artworks. 


Augmented Text Company The company which produces the Author software, Reader 


software and Liquid software. 
Director is Frode Hegland. 
https://www.augmentedtext.info 


#Institution. Timeline: 9 March 2016- 


Author means in the context of this work someone who writes a document, which includes 
academic papers and books. Can also refer to the Author software by The Augmented Text 
Company. 

Etymology: “mid-14c., auctor, autour, autor “father, creator, one who brings about, one who 
makes or creates” someone or something, from Old French auctor, acteor “author, originator, 
creator, instigator” (12c., Modern French auteur) and directly from Latin auctor “promoter, 
producer, father, progenitor; builder, founder; trustworthy writer, authority; historian; 
performer, doer; responsible person, teacher,” literally “one who causes to grow,” agent noun 


from auctus, past participle of augere “to increase,” from PIE root *aug- (1) “to increase. 


From https://www.etymonline.com/search?q=Author 


Barbara Tversky Guest presenter at the Future Text Lab. 

Issue 1.1 on the 21st of January 2022. 

Professor emerita of psychology at Stanford University and a professor of psychology and 
education at Teachers College, Columbia University.Tversky specialises in cognitive 
psychology. 

Author of Mind in Motion: How Action Shapes Thought. Basic Books, 2019. 


#person 


Benediktine Cyberspace A 3-D visualization inspired by Michael Benedikt’s seminal text, 


“Cyberspace: First Steps” 


BibTeX is a specific format for conveying citation information within the LaTex 
environment, developed by Oren Patashnik and Leslie Lamport, released in 1985. The benefit 
of the system was to separate citation information from presentation style and it is human 
readable, though it slightly looks like code. It inspired the format of Visual-Meta and Visual- 


Meta contains a straight BibTeX section to allow the document which contains it to be cited. 


Bob Horn, Robert E. Horn is a political scientist with a special interest in policy 
communication, social and organizational learning, and knowledge management (especially 
in sustainability and national security affairs), known for producing large scale murals. 

Was Senior Researcher at Stanford University's Human Science and Technology Advanced 
Research Institute (H-STAR) for 27 years. While he was a research associate at Columbia 
University, he created a widely used methodology for the analysis of any complex stable 
subject matter. This research became an international consulting company, Information 
Mapping, Inc. He was CEO and chairman for 20 years. He has taught at Harvard, Columbia, 
and Sheffield (UK) universities. Recently, he has been developing large info-murals and 


leading “mess mapping” projects and workshops to enable decision making groups get their 


minds around larger contexts for strategic discussions of wicked problems and mega-messes. 
The projects range from global climate change, energy security, nuclear waste disposal, 
NASA’s research programs. He was the synthesizer and visualizer for the World Business 
Council for Sustainable Development’s Vision 2050 project the European-Commission- 
supported project on policy options for a resource efficient Europe (POLFREE). His 
development of visual argumentation mapping has resulted in the publication of the Mapping 
Great Debates series, which, in the same year received a full-page review in Nature, as well 
as being hung as artwork in a national museum in The Hague as part of an exhibit on 
information design as a fine art. 

His 7 books include Visual Language: Global Communication for the 21st Century (1998) 
and Mapping Hypertext (1989, a book about the web 2 years before the web was created). 
His most recent book (still in draft) is: The Little Book of Wicked Problems and Social 
Messes. 

He has received Lifetime Achievement Awards from the Association of Computing 
Machinery (ACM) and the International Society for Performance and Instruction. He is a 


fellow of the World Academy of Art and Science. 
www.bobhorn.us 


#person. Timeline: - Lives in: California, USA 


Bob Horn Robert E. “Bob” Horn is a fellow of the World Academy of Art and Science. For 
27 years, he was a Senior Researcher the Human Science and Technology Advanced 
Research Institute (H-STAR), Stanford University, where he worked with international task 
forces and governments on wicked problems an social messes. He has taught at Harvard and 


Columbia universities and is the author/editor of ten books. 
He produces large scale murals. 


www.bobhorn.us 


Bob Stein is the director of the Institute for the Future of the Book 


He founded The Voyager Company in 1985, the first commercial multimedia CD-ROM 
publisher, and The Criterion Collection in 1984, a collection of definitive films on digital 
media with in-depth background information (including the first films with recorded audio 
commentary). Worked with Alan Kay at the Atari Research Group on various electronic 


publishing projects. Currently works on The Tapestry Project. 
#person. Timeline: 20 April 1946- 


Book a published work, on any substrate, inclduing paper or digital. 
Generally longer than a ‘paper’ but in terms of digital work, there is little technical 


difference, the difference is mostly social. 


Brandel Zachernuk AR / VR Creative technologist by day, digitally-embodied cognition 
enthusiast by night! 

Something that I've been very passionate about trying to investigate and play with over the 
last 10 years or so is what is the most pedestrian thing that you can do with virtual reality and 
a technology that was was word processing was reading and reading and thinking about what 
are the basic building blocks of that process of writing and reading that can fundamentally be 
changed by buying virtual reality? Realising that if you don't have a screen, you have the 
ability for information to mean what it means for your purposes, rather than the technical 
limitations that apply as a consequence of a mouse or keyboard or things like that, and also 
deeply invested in understanding some of the emerging cognitive science and and 
neurophysiological sort of views about what it is that the mind is and the way that we work 
best. So reading about learning about what people call for in cognition, it's embedded, 
embodied in active and extended mind and how that might pertain to what we should be 
doing with software and systems, both as well as hardware if necessary to make it so that we 


can. We can think properly and express properly and stuff. 
https://youtu.be/2NcSCOrVw24?t=879 


https://twitter.com/zachernuk 


https://www.youtube.com/channel/UCOP9cT-IwB6oxAcfvS1x_ZQ 
https://futuretextlab.info/tag/bybrandel/ 
http://www.zachernuk.com/ 


#Person. Timeline: -. Lives in: California, USA. 


Brett Jackson VR developer. 
Currently working on Idea Engine (https://linktr.ee/ideaengine), an interactive mind map. 


Creator of Dimensional, Jigsaw360, Breath Tech. 


https://twitter.com/JumbliVR 


Caitlin Fisher is professor and Chair, Cinema and Media Arts and Director, Augmented 
Reality Lab, Director, Immersive Storytelling Lab, School of the Arts, Media, Performance 
and Design, York University 


President, Electronic Literature Organization 
Works with AR and VR and electronic literature, future cinema, emerging technology. 


#person. Timeline: - 


Canada https://en.wikipedia.org/wiki/Canada 


#Location 


Chalktalk Chalktalk is a digital presentation and communication language in development at 
New York University's Future Reality Lab. Using a blackboard-like interface, it allows a 
presenter to create and interact with animated digital sketches in order to demonstrate ideas 


and concepts in the context of a live presentation or conversation. 


Charlie Hargood Academic in Games Technology at Bournemouth University. 


Christopher Gutteridge is also a contributor to the first volume of The Future of Text. 
Lives: Southampton, UK. 


Works at the University of Southampton. 


citation In this work, the term’ citation’ is both the ‘citation’ mention in the body of the 
document, and its corresponding ‘references’ entry for the source material in the Reference 
Appendix at the end of the document, though for the sake of clarity I will mainly use 
‘citation’ to refer to the in-body half and ‘reference’ for the entry of the source in the 


References section. 


#term 


Claus Atzenbeck is a cocurator of The Future of Text Symposium. 

He holds a professor position at Hof University, Germany and is leading the Visual Analytics 
research group at the university’s Institute of Information Systems. 

He served as general co-chair of the 30th Anniversary ACM Hypertext Conference 2019 in 
Hof, Germany (https://human.tisys.de/ht2019/, 7 @ACMHT), is co-organizer of the 
HUMAN workshop series (https://human.1isys.de/human/, 7 @HUMAN_HT), and the 
initiator of the Historic Hypertext Project (https://human.iisys.de/hist_ HT/, 7 @hist_HT). 
http://www.atzenbeck.de https://twitter.com/clausatz https://www.iisys.de 

#person. Timeline: . Born: Germany. Lives in: Germany. Research Interests: Hypermedia 
including spatial and navigational structures, spatial and temporal parsing for spatial 
hypertexts, and hypertext narratives. A specific focus of his work is on intelligent user 


interfaces for visual analytics. 


Co-Evolution Engelbart Concept: Most capabilities are improved, or augmented, by many 


interdependent technical and non-technical elements, of which tools make up only a small 


part: On one hand, there is the human system, which includes paradigms, organizations, 
procedures, customs, methods, language, attitudes, skills, knowledge, training and so on- all 
of which all exists within the basic perceptual and motor capabilities of the human being. On 
the other hand, there is the tool system, which includes media, computers, communications 


systems etc. Together, they comprise the augmentation system. 


#dougconcept #term 


Cognition “is the mental action or process of acquiring knowledge and understanding 


through thought, experience, and the senses”. 


https://en.wikipedia.org/wiki/Cognition 


colophon “In publishing, a colophon (/‘kvlofon, -fon/)[1] is a brief statement containing 
information about the publication of a book such as the place of publication, the publisher, 


and the date of publication.” 
https://en.wikipedia.org/wiki/Colophon_ (publishing) 
Orignally, in Mesopotamia, colophons contained quite a bit more information: 


https://colophons-and-scholars.com/home 


colophon A publisher's emblem or imprint, usually on the title page of a book which inspired 


Visual-Meta, which is similar data (in BibTeX form), at the back of a document. 


“In publishing, a colophon (/‘kolofon, -fon/)[1] is a brief statement containing information 
about the publication of a book such as the place of publication, the publisher, and the date of 


publication.” 

https://en.wikipedia.org/wiki/Colophon_(publishing) 

Originally, in Mesopotamia, colophons contained quite a bit more information: 
https://colophons-and-scholars.com/home 


A form of metadata. 


computational text Vint Cerf likes to use the term ‘computational text’ to mean text which 
can be interacted with computationally. 


Relates to encoding in Visual-Meta. 


concept, defined concept, defined concepts is, in the context of this book, any text which is 
defined by an author. 

When using the Augmented Text Tools Author this is exported in a PDF as a Glossary which 
becomes interactive when opened in Reader. In Author, The Map view uses the definition to 
draw lines where text from a definition is also present on the Map. When a document is 
exported to PDF the Defined Concepts become Glossary Terms. 

This is as opposed to inferred concept. 

Also aView in Author and in Reader (with Visual-Meta) to show all named entities (and 
headings). 


#term 


CRS (Commercial Resupply Services) are a series of flights awarded by NASA for the 
delivery of cargo and supplies to the International Space Station (ISS) on commercially 


operated spacecraft. 


https://en.wikipedia.org/wiki/Commercial Resupply Services 


Cyberspace “The term "cyberspace" first appeared in the visual arts in the late 1960s, when 
Danish artist Susanne Ussing (1940-1998) and her partner architect Carsten Hoff (b. 1934) 
constituted themselves as Atelier Cyberspace.” Used in this context to mean a digital world, 


not necessarily in VR or AR. 


Cynthia Haynes is former Director of Rhetorics, Communication, and Information Design 


Ph.D program and currently Professor of English at Clemson University. 

Her research interests are rhetoric, composition, multimodal pedagogy, virtual worlds, VR, 
critical theory, computer games studies, and the rhetoric of war and terrorism. Her recent 
book, The Homesick Phone Book: Addressing Rhetoric in the Age of Perpetual 

Conflict (Southern Illinois University Press, 2016) won the 2017 Rhetoric Society of America 
annual book prize. She is currently working on a book, Unalienable Rites: The Architecture 
of Mass Rhetoric. 

She is, along with Jan Holmevik, member of the Kairos Editorial Board and co-founders of 
the journal's electronic partner LinguaMOO. 


#person. Timeline: -. Lives in: Clemson, South Carolina, USA. 


DALL-E DALL-E (stylized as DALL-E) and DALL-E 2 are machine learning models 


developed by OpenAI to generate digital images from natural language descriptions. 


The name is a portmanteau of the names of animated robot Pixar character WALL-E and the 


Spanish surrealist artist Salvador Dali. 
https://en.wikipedia.org/wiki/DALL-E https://openai.com/blog/dall-e/ 


#technology. AI. Machine Learning. ML. 


Dame Wendy Hall, Wendy Hall Frode Hegland’s PhD primary supervisor, along with Les 
Carr and David Millard at the University of Southampton. 


Daniel Dennett is an American philosopher, writer, and cognitive scientist whose research 
centers on the philosophy of mind, philosophy of science, and philosophy of biology, 
particularly as those fields relate to evolutionary biology and cognitive science. 
https://en.wikipedia.org/wiki/Daniel_ Dennett 

#person Timeline: 28 March 1942- 


David Jay Bolter Wesley Chair in New Media, Georgia Tech and author of The Digital 
Plenitude: The Decline of Elite Culture and the Rise of New Media. 


Along with John B. Smith and Michael Joyce, Bolter co-created Storyspace, a software 
program for creating, editing, and displaying hypertext fiction. 
Contributor to The Future of Text Volume 1. 


Timeline: 17 August 1951-. Born: USA 


David Millard Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and Les Carr 


at the University of Southampton. 


Deena Larsen is a new media and hypertext author involved in the creative electronic 


writing community since the 1980s. From USA. 
https://en.wikipedia.org/wiki/Deena_Larsen 


#person. Timeline: 1964- 


Definition when it comes to glossaries, it is the user’s stated meaning of a term. 


Dene Grigar is a cocurator of The Future of Text Symposium. 

PhD, Director of the Electronic Literature Lab; Managing Director and Curator of ELO’s The 
NEXT. 

Digital artist and scholar based in Vancouver, Washington. Professor and Director of the 
Creative Media & Digital Culture Program at Washington State University Vancouver, USA. 
A prioneer of digital literature. Former president of the Electronic Literature Organisation. 


#person. Timeline: -. Interests: Electronic Literature. 


Distance-Independent Millimeters, DMM Google researchers needed a way to design 


screens that could be easily read at any distance. To help solve this, they came up with 


"Distance-Independent Millimeters" or DMMs. A VR and AR term. 
More detail at: https://www.youtube.com/watch?v=ES9j;ArHRFHQ 


#term 


DOI ‘Document Object Identifiers’ 
An effort to make addressing academic documents via the web more robust. 


Used in Author software to let the user paste a DOI to cite an academic document which is 


then sent to CrossRef to be parsed into BibTeX which is then used to create a full citation. 


#Infrastructure 


Doug Engelbart, Douglas Carl Engelbart, Doug From Wikipedia: “He was an engineer and 
inventor, and an early computer and Internet pioneer. He is best known for his work on 
founding the field of human-computer interaction, particularly while at his Augmentation 
Research Center Lab in SRI International, which resulted in creation of the computer mouse, 
and the development of hypertext, networked computers, and precursors to graphical user 
interfaces. These were demonstrated at The Mother of All Demos in 1968. Engelbart’s law, 
the observation that the intrinsic rate of human performance is exponential, is named after 
him.” 

His seminal paper was ‘Augmenting Human Intellect’, 1962. 

He was also my mentor and greatly influened my work, resulting in my company called The 
Augmented Text Compant and my word processor being called Author, in honour of his 


‘Augment’ system. Visual-Meta is inspired by his Open Hyperdocument work. 


#person. Timeline: 30 January 1925- 2 July 2013 


Doug Engelbart “He was an engineer and inventor, and an early computer and Internet 
pioneer. He is best known for his work on founding the field of human-computer interaction, 


particularly while at his Augmentation Research Center Lab in SRI International, which 


resulted in creation of the computer mouse, and the development of hypertext, networked 
computers, and precursors to graphical user interfaces. These were demonstrated at The 
Mother of All Demos in 1968. Engelbart's law, the observation that the intrinsic rate of 


human performance is exponential, is named after him.” 
https://en.wikipedia.org/wiki/Douglas_Engelbart 


He was also my mentor and greatly influened my work, resulting in my company called The 
Augmented Text Compant and my word processor being called Author, in honour of his 


‘Augment’ system. Visual-Meta is inspired by his Open Hyperdocument work. 


Eduardo Kac is internationally recognized for his groundbreaking work in contemporary art 
and poetry. 

In the early 1980s, Kac created digital, holographic and online works that anticipated the 
global culture we live in today, composed of ever-changing information in constant flux. In 
1997 the artist coined the term "Bio Art," igniting the development of this new art form with 
works such as his transgenic rabbit GFP Bunny (2000) and Natural History of the Enigma 
(2009), which earned him the Golden Nica, the most prestigious award in the field of media 
art. GFP Bunny has become a global phenomenon, having been appropriated by major 
popular culture franchises such as Sherlock, Big Bang Theory and Simpsons, and by writers 
such as Margaret Atwood and Michael Crichton. In 2017, Kac created Inner Telescope, a 
work conceived for and realized in outer space with the cooperation of French astronaut 
Thomas Pesquet. Kac’s singular and highly influential career spans poetry, performance, 
drawing, printmaking, photography, artist's books, early digital and online works, holography, 
telepresence, bio art, and space art. Kac has also authored or edited several books, including 
Telepresence and Bio Art -- Networking Humans, Rabbits and Robots (University of 
Michigan Press, 2005). Kac’s work has been exhibited internationally at venues such as New 
Museum, New York; Pompidou Center, Paris; MAXXI-Museum of XXI Century Arts, Rome; 
Mori Art Museum, Tokyo; Reina Sofia Museum, Madrid; Power Station of Art, Shanghai; 
and Seoul Museum of Art, Korea. Kac's work has been showcased in biennials such as Venice 


Biennale, Italy; Yokohama Triennial, Japan; Gwangju Biennale, Korea; Bienal de Sao Paulo, 


Brazil; and Bienal de Habana, Cuba. His works are in major collections such as Museum of 
Modern Art-MoMA, New York; Tate Modern, London; Victoria & Albert Museum, London; 
Museum Les Abattoirs—Frac Occitanie Toulouse, France; Valencian Institute of Modern Art- 
IVAM, Spain; Museum ZKM, Karlsruhe, Germany; and Museum of Contemporary Art of 
Sao Paulo, among others. Kac was elected as full member to the IAF (International 
Astronautical Federation) Technical Activities Committee for the Cultural Utilisation of 


Space (ITACCUS). 


#person. Timeline: -. 1962 -. Born: Brazil. Lives in: USA. 


Egypt https://en.wikipedia.org/wiki/Egypt 


#Location 


Electronic Literature “Electronic literature or digital literature is a genre of literature 
encompassing works created exclusively on and for digital devices, such as computers, 
tablets, and mobile phones. A work of electronic literature can be defined as "a construction 
whose literary aesthetics emerge from computation", "work that could only exist in the space 
for which it was developed/written/coded—the digital space”.” https://en.wikipedia.org/wiki/ 


Electronic_literature 


ESA (European Space Agency) is an intergovernmental organisation of 22 member states 


dedicated to the exploration of space. 


https://en.wikipedia.org/wiki/European_Space_Agency 


Extrinsic Attributes of an object that are represented visually though that object’s spatial 


position in a coordinate system represening the domain of their possible values. 


Fabien Benetou Prototyping - European Parliament Innovation lab WebXR consultant - 


Former UNICEF Innovation Fund WebXR technical advisor. VR prototypist. 
https://twitter.com/utopiah https://fabien.benetou.fr 


#person. Timeline: 6 November 1982- 


Fermat is a spatial canvas where every element is programmable by the end user. Users can 


create their own tools and share them with every other user via a public toolbox. 


Recently, we discovered that the most transformative tools built on top of Fermat were using 


Artificial Intelligence. 
(We're bringing back HyperCard!). A tool for thought. 
https://fermat.ws 


#Software. Timeline: -. Keywords: AI. GPT-3. Stable Diffusion. 


flatland a semi-humorous term we use to refer to traditional displays, as opposed to the 
augmented environments of VR and AR. We feel that it is important to be able to move data, 


including metadata between these environments. 


Folding Folding of a document into a table of contents is enabled through Visual-Meta. 


Frode Hegland, Frode Alexander Hegland is cohost of the Future of Text Symposium. 


Director of The Augmented Text Company where he designed the macOS Author word 


processor, Reader PDF viewer and the Liquid software. https://www.augmentedtext.info/ 
Editor of the 'The Future of Text' series of books and The Future of Text Journal. 

Designed Visual-Meta. 

His mentor was Doug Engelbart. Greatly influenced by his friend Ted Nelson. 

PhD from the University of Southampton. Advisors Wendy Hall, Les Carr and David Millard. 


#person. #Timeline: Born 2 June 1968-. Born in: Bergen, Norway. Lives in: London, UK. 


Future Reality Lab New York University's Future Reality Lab. 


https://frl.nyu.edu 


Future Text Lab a group of people who meet to work on the future of text. Active members 
currently include Vint Cerf, Adam Wern, Fabien Benetou, Bob Horn, Alan Laidlaw, Mark 
Anderson, Peter Wasilko and Frode Hegland. 


Future Text Lab A lab dedicated to studying how working with knowledge, particularly text, 
can be done effectively in VR/AR/Metaverse. 


We host Guest presenters and publish a Journal which is part of The Future of Text series. 
https://futuretextlab.info 


#Institution. Timeline: - 2021 


Gavin Menichini Guest presenter at the Future Text Lab. Issue 1.2 on the 25th of February 
2022. 


Strategic Account Executive at the VR company Immersed. “Work Faster in VR” 


https://immersed.com 


Gems software developed by Lorenzo Bernaschina. 


https://gemsnotes.app/ 


George Lakoff is an American cognitive linguist and philosopher, best known for his thesis 
that people's lives are significantly influenced by the conceptual metaphors they use to 


explain complex phenomena. 


https://en.wikipedia.org/wiki/George_Lakoff 
#person Timeline: 24 May 1941 


Germany https://en.wikipedia.org/wiki/Germany 


#Location 


Glossary means, in the context of this work, a specific user or editor list of definitions for a 


specific document. 
Defined Concepts in the Author Software is exported as a Glossary. 
This is different from a dictionary since dictionary definitions have general validity. 


Inspired by discussions with Doug Engelbart. 


Glossary In the context of my work and thinking, a glossary is a set of Defined Concepts 


which an author has created and which is then exported as a Glossary for the reader. 


The primary purpose of this is for the author to have to think through their writing by 
explicitly stating what something is. The definition of the defined concept can then include 
text which also has a definition and in the Map view in Author this connection can be shown 


as a line when either defined term is selected. 


The secondary purpose is for the author to edit this Glossary to make sure it is coherent for 


export and it can then help a reader understand the author’s intentions. 


#term 


Glossing A way of elucidating parts of text. 


https://www.etymonline.com/search?q=gloss 


Google “is an American multinational technology company that focuses on search engine 


technology, online advertising, cloud computing, computer software, quantum computing, e- 
commerce, artificial intelligence, and consumer electronics.” https://en.wikipedia.org/wiki/ 
Google 


#Institution. Timeline: 1998-. USA 


GPT “Generative Pre-trained Transformer” inclduinig GPT-2 and GPT-3. 
https://en.wikipedia.org/wiki/GPT-2 

https://en.wikipedia.org/wiki/GPT-3 

Product of OpenAI. 


GPT-3 ‘Generative Pre-trained Transformer 3’ (GPT-3; stylized GPT-3) is an autoregressive 
language model that uses deep learning to produce human-like text. https://en.wikipedia.org/ 
wiki/GPT-3 


“In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to 


wow 


generate computer code, poetry, and prose is not just "amazing", "spooky", and "humbling", 
but also "more than a little terrifying". https://www.worldcat.org/issn/0362-433 1 


Timeline: 11 June 2020-. Keywords: Deep Learning. AI. ML. Machine Learning. 


HMD Head Mounted Display for use in VR or AR. 


#Hardware | Timeline: 


HTML “The HyperText Markup Language or HTML is the standard markup language for 
documents designed to be displayed in a web browser.” https://en.wikipedia.org/wiki/HTML 


A technology we are also using, along with PDF. 


hypertext a term invented by Ted Nelson for interactive and connected digital text. 


Immersed Immersed is a virtual reality product, working productivity software for virtual 


offices. 


https://immersed.com 


Intrinsic Attribute of a data object that are not represented by that object’s spatial position in 
a given visualization but which may be represented by its size, shape, color, transparence, or 


other positionally independent visualization technique. 


Ismail Serageldin is a long time collaborator of the Future of Text Symposium. 


Founding Director of the Bibliotheca Alexandrina (BA) and was Vice President of the World 
Bank. 


#person. Timeline: 1944-. Born: Egypt. 


ISS (International Space Station) is the largest modular space station currently in low Earth 


orbit. 


https://en.wikipedia.org/wiki/International_Space_ Station 


Jack Kausch is an academic whose research interests are in linguistics, ontologies and the 


semantic web, embeddings and the history of writing. 

He is a PhD student at the Western University of Ontario in Canada. 
His advisor is Kamran Sedig. 

He works as a data labeller for OpenAI and is interested in AI. 


#person. Timeline: 14 October 1993-. Born in: Ann Arbor, Michigan. Lives in: Ontario, 


Canada. 


Jacob Hazelgrove is the programmer for the Augmented Text Company for Frode Hegland, 


including Author and Reader, as well as imlementor of Visual-Meta export from Author and 


import and interaction in Reader. 


#Person | Timeline: - 


Jad Esber Guest presenter at the Future Text Lab. 
CEO of koodos and an Affiliate at the Berkman Klein Centre for Internet & Society at 
Harvard. 


https://twitter.com/Jad_AE 


Jan Holmevik, Jan Rune Holmevik is Associate Professor of English and Special Advisor 
to the Vice President for Information Technology & CIO, as well as Past President of the 
Faculty Senate at Clemson University. 

PhD in Humanistic Informatics from the University of Bergen, Norway and a Master’s 
degree in the history of technology from the Norwegian University of Science and 
Technology in Trondheim, Norway. 

He specializes in Interactive and Social Media and conducts research in game design, game 
culture, digital literacy, social media, visual communication, humanistic informatics, 
information design, data visualization, social media forensics, transmedia, and electracy. 
http://t.co/70 V4voInkm https://twitter.com/holmevik 


#person. Timeline: - 


Jaron Lanier “American computer scientist, visual artist, computer philosophy writer, 


technologist, futurist, and composer of contemporary classical music. 


Considered a founder of the field of virtual reality.” https://en.wikipedia.org/wiki/ 


Jaron_Lanier 
VR pioneer. 
#Person. Timeline: 3 May 1960- . Born: NY USA 


Jim Strahorn Contributor to the Future Text Lab. 


Johannah Rodgers Author, ‘Engineering Language: Teaching Machines to Read and Write 
in the U.S. 1869 - 1969. 


Contributor to The Future of Text volume 1. 


Jonathan Finn is a developer of creativity apps (e.g. Sibelius music writing system). 
Contributor to The Future of Text volume 2 and editing for volume 3. 


timeline: - 


Jorge Luis Borges “was an Argentinian writer, essayist and translator known for his 
trademark themes: dreams, labyrinths, libraries, language and mythology. His stories, non- 
linear narratives that mix fact, fantasy, hox and forgery, are generally considered to have 
reinvented modern literature. This dialogue speculates on Borges’s position regarding 
language and virtual reality based on his short stories Tl6n, Uqbar, Orbis Tertius and Funes, 
the Memorious. Moreover, the entire conversation makes use of many of Borges’s literary 
techniques. Most of the time I stay close to what the main characters could have plausibly 
said in such a situation, but, like Borges in his own stories, I also diverge from that and use 
the two characters to purse my own arguments. Hinted at by the fact that the footage was 


recorded in Borges’s headset, this is the kind of thing he would write.“ 
https://en.wikipedia.org/wiki/Jorge_ Luis Borges 
Timeline: 1899 — 1986. Born Argentina. 


Kalev Leetaru is an American internet entrepreneur, academic, and senior fellow at the 
George Washington University School of Engineering and Applied Science Center for Cyber 
& Homeland Security in Washington, D.C. USA. 


Best known for his role as the co-creator of the Global Database of Events, Language, and 
Tone (GDELT) with Philip Schrodt: "an initiative to construct a catalog of human societal- 
scale behavior and beliefs across all countries of the world, connecting every person, 
organization, location, count, theme, news source, and event across the planet into a single 
massive network that captures what's happening around the world, what its context is and 
who's involved, and how the world is feeling about it, every single day.” 
https://en.wikipedia.org/wiki/GDELT Project https://blog.gdeltproject.org/web- 
summit-2017-video/ 


#person. Timeline: -. 


Ken Perlin, Kenneth H. Perlin is a professor in the Department of Computer Science at 
New York University, founding director of the Media Research Lab at NYU, director of the 
Future Reality Lab at NYU, and the Director of the Games for Learning Institute. 

B.A. degree in Theoretical Mathematics from Harvard University. M.S. and PhD in 
Computer Science from the Courant Institute of Mathematical Sciences, New York 
University. from the same institution. He developed or was involved with the development of 
techniques such as Perlin noise, real-time interactive character animation, and computer-user 
interfaces. He is best known for the development of Perlin noise and Simplex noise, both of 
which are algorithms for realistic-looking Gradient noise. “Received an Academy Award for 
the development of Perlin noise. He had introduced this technique with the goal to produce 
natural-appearing textures on computer-generated surfaces for motion picture visual effects, 
while working on the Walt Disney Productions' 1982 feature film TRON for which he had 
developed a large part of the software.” https://en.wikipedia.org/wiki/Ken_Perlin 

Research interests include graphics, animation, multimedia and science education, VR and 


AR. 


#person. Timeline: - 


Les Carr, Leslie Carr Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and 


David Millard at the University of Southampton. 


Lev Vygotsky was a Russian cognitive scientist, psychologist, constructivist and critical 
realist whose work focused on the internal mental structure of an individual. 
Methodologically, he focused on relationships, processes and levels of analysis. He is best 
known for sociocultural theory, a developmental school of thought focused on the 
relationship between thought and language as independent and dynamic processes in 
ontogenesis, phylogenesis, and within a cultural context. This dialogue speculates on 
Vygotsky’s position regarding language and virtual reality based on his book Thought and 
Language. 


Timeline: 1896 — 1934. Born: Russia. 


Liquid Information, liquid Frode Hegland’s philosophy of interactive computing. 


Developed while a student in New York, the fundamentals are removing barriers to rich 
interaction. Later influenced and expanded by the work of his mentor Doug Engelbart and 


fleshed out in discussions with Sarah Walton. 


https://www.liquidinformation.org 


LiSA Software by Frode Hegland, featuring the voice of Janine Earl. 


Discontinued: http://www. liquid.info/lisa/ 


Livia Polanyi is also a contributor to the first volume of The Future of Text. 


Lives: NY, USA. 


Lorenzo Bernaschina is a software engineer. 


Founder of Gems, a PKM SaaS to visually manage notes with the help of AI. Interested in 


tools for thought, human-computer interaction, and everything related to mind- expanding 


technology. 


Master’s degree in Machine Learning and Artificial Intelligence from Politecnico di Milano. 


Full-stack developer (MEAN/MERN stack + iOS). 


#person. Timeline: 3 July 1995-. From Lake Como, Italy. 


Machine learning Machine learning (ML) is a field of inquiry devoted to understanding and 
building methods that 'learn’, that is, methods that leverage data to improve performance on 


some set of tasks. 
A type of Artificial Intelligence (AI). 
https://en.wikipedia.org/wiki/Machine_learning 


#Technology. Timeline: 20th century-. 


manuscript the authoring format, such as Microsoft Word, which is then either shared as-is, 


and stays editable, or is exported to be published in a publish format, such as PDF. 


#term 


Map In this context, a view in the Author software. 


Here the user can place text anywhere they want. If there are defined concepts on the map, 
the user can click on them and lines will emanate to any text on the map which is in that 


text’s definition. 


#term 


Mark Anderson Independent researcher in Hypertext and Knowledge systems. Associated 


with the Web & Internet Science (WAIS) Lab at Southampton University. 
Mark was heavily involved in the technical specification and ACM launch of Visual-Meta. 


Mark has a background in organisational structure, knowledge, process change and data 


interchange. 


Co-Editor of the Journal. 


Mark Anderson is a cocurator of The Future of Text Symposium. 

Independent researcher and consultant in hypertext, knowledge systems, and the retention/ 
support of organisational knowledge. A background in organisational structure, knowledge, 
process change and data interchange consulting across public, private and NFP sectors. 

A long-term contributor to open projects and online communities since the mid-90s. Also a 
contributor to various data formats: PDF metadata, IPTC v4 and most recently the technical 
specification of Visual-Meta including facilitating ACM’s implementation of Visual-Meta. 
Part of hypertext study has included the recovery for use of early hypertext systems otherwise 
lost to current researchers and creating ebooks for some of Ted Nelson’s key (out of print) 
books. 

Publications & datasets: https://scholar.google.co.uk/citations?hl=en&user=jn8crEAAAAAJ 
Other public resources: https://www.acrobatfaq.com 

PhD in Web Science from the University of Southampton: advisors Les Carr and Dave 
Millard A Visiting Fellow at the University of Southampton, associated with its Web & 
Internet Science (WAIS) Lab. 


#person. Timeline: 1959 -. Lives in: Portsmouth, UK. 


Marvin Minsky was an American cognitive and computer scientist concerned largely with 
research of artificial intelligence (AI), co-founder of the Massachusetts Institute of 
Technology's AI laboratory, and author of several texts concerning AI and philosophy. 
https://en.wikipedia.org/wiki/Marvin_ Minsky 

#person Timeline: 9 August 1927 — 24 January 2016 


Mesopotamia a historical region within the Tigris—Euphrates river system in the northern 
part of the Fertile Crescent where early writing developed (in parallel to, or influenced by/ 


influencing Egyptian writing), including the use of colophons. 


Meta is a large social media company which is focusing on what they term the ‘metaverse’. 
Formerly called Facebook. 
https://about.facebook.com 


#Institution. Timeline: 4 February 2004-. 


metadata information about other information, in the case of documents, this can include 


structural information (headings for example), biblio. 


Metaverse From Wikipedia: “A metaverse is a network of 3D virtual worlds focused on 
social connection” 

“The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow 
Crash” 

https://en.wikipedia.org/wiki/Metaverse 

At the time of writing, 2022, the term has been popularised by Meta, the company previously 
called Facebook. Most often used in conjunction with AR/VR augmented environments, not 


such much the 3D worlds where the user access the world through a flat screen. 


Mez Breeze is an Expert and Innovator of contemporary digital culture. 

Since the early 1990’s, Mez Breeze has published over 300 seminal works including award- 
winning electronic based writing, Virtual Reality literature, Artificial Intelligence artworks/ 
projects, books, games, and other genre-defying output all while teaching and supporting 
digital art and electronic literature. In April 2022, Mez’s Artificial Intelligence Artwork ‘Post 
Glee[son] - Outside [R]’ (created through digitally stitching GauGAN and VQGAN+CLIP 
output) made the finals of the 2022 Goulburn Art Award. In July 2019, Mez won the 2019 
Marjorie C. Luesebrink Career Achievement Award which: “...honors a visionary artist and/ 


or scholar who has brought excellence to the field of electronic literature and has inspired 


others to help create and build the field.” Mez's projects are taught worldwide with her works 
residing in Collections as diverse as The World Bank, Cornell's Rose Goldsen Archive and 
the National Library of Australia. She currently serves as an Advisor to the Mixed 
Augmented Reality Art Research Organisation, an Editorial Board Member of the Digital 
Journal Thresholds, and is a Senior Research Affiliate of the Humanities and Critical Code 
Studies Lab. 

Her latest book ‘[Por]TrAIts: AI Characters + Their Microstories’ (written in her trademark 
Mezangelle style while incorporating AI output) was a Top Seller on the publishing platform 
Itchio in September 2022. 

https://www.mezbreezedesign.com, https://mezbreeze.itch.io/portraits-volume-one 

#person. Timeline: 1970-. Company: Mez Breeze Design. Lives in: Australia. Key Interests: 


AI. VR. XR. 


Michael Roberts is a Scientist Mike Roberts currently resides in the Santa Cruz Mountains, 
California, USA. 

He received his doctorate in the early 90’s, working on the intersection between parallel 
computing and no-code visual programming, specifically defining an isomorphism between 
Tony Hoare’s Communicating Sequential Processes (CSP) calculus and graph theory. 

After moving the US, he began work on tools for interactive, creative, computer-based 
media, specifically targeting computer graphics, multimedia, augmented and virtual realities. 
His work has influenced the technical and business direction of many US-based fortune 500 
companies including Xerox, Autodesk, Progress Software, IPS/Motorola as well as prominent 
Japanese companies like DNP. 

Prior to founding ConstuctiveLabs, his current metaverse company, he recently spent 9.5 
years at renowned research lab Xerox PARC, where he worked on projects including 
conversational agents, contextual and artificial intelligence, and AR / VR. His interests in 
tool-making for creatives are informed by studies in traditional tool ecosystems and slow 
food production. 


Founder of Constructivelabs. 


#person. 


Murals, mural This is a special entry to experiment with layouts. 


NIC Doug Engelbart concept: Networked Improvement Community “Consider an 
"Improvement Community" (IC) as collectively engaged in improving an agreed-upon set 
either of individual capabilities, or of collective group capabilities-e.g. a professional society. 
Let's introduce a new category, a "Networked Improvement Community" (NIC): an IC that is 
consciously and effectively employing best-possible DKR (Dynamic Knowledge Repository) 
development and usage.” 


(augmenting society's collective IQ). 


NLS From Wikipedia: “NLS, or the "oN-Line System", was a revolutionary computer 
collaboration system developed in the 1960s. Designed by Douglas Engelbart and 
implemented by researchers at the Augmentation Research Center (ARC) at the Stanford 
Research Institute (SRI), the NLS system was the first to employ the practical use of 
hypertext links, the mouse, raster-scan video monitors, information organized by relevance, 
screen windowing, presentation programs, and other modern computing concepts. It was 
funded by ARPA (the predecessor to Defense Advanced Research Projects Agency), NASA, 
and the US Air Force.” https://en.wikipedia.org/wiki/NLS_(computer_system) 


SRI sold NLS to Tymshare in 1977 and renamed it Augment. 


Norway https://en.wikipedia.org/wiki/Norway 


#Location 


NYU (New York University) is a private research university in New York City. Chartered in 
1831 by the New York State Legislature, NYU was founded by a group of New Yorkers led 


by then-Secretary of the Treasury Albert Gallatin. 


https://en.wikipedia.org/wiki/New_ York University 


Omar Rizwan Contributor to the Journal. 


Omar has been interested in new computer interfaces and new ways of programming (aren't 


these the same thing?). He has worked at Dynamicland, at Stripe, and at Khan Academy. 


"; am determined to move beyond this way of interacting with systems" 


Among other things, I'm the creator of s.vScreenotate, a tool for macOS and Windows which 
captures the text and origin (URL, window title, ...) whenever you take a screenshot. 


https://screenotate.com/ 


https://twitter.com/rsnous 


https://omar.website 


Open AIT is an artificial intelligence (AI) research laboratory consisting of the for-profit 


corporation OpenAI LP and its parent company, the non-profit OpenAI Inc. 
Products: DALL-E, GPT-3. GPT-2. OpenAI Gym 

https://openai.com/ 

https://en.wikipedia.org/wiki/OpenAI 


OpenAT is an artificial intelligence (AI) research laboratory consisting of the for-profit 


corporation OpenAI LP and its parent company, the non-profit OpenAI Inc. 
Products: GPT-1. GPT-2. GPT-3 
https://openai.com 


https://en.wikipedia.org/wiki/OpenAI 


OpenAL is an artificial intelligence (AI) research laboratory. 


“,.consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit 
OpenAI Inc. 

The company, considered a competitor to DeepMind, conducts research in the field of AI 
with the stated goal of promoting and developing friendly AI in a way that benefits humanity 
as a whole. The organization was founded in San Francisco in late 2015 by Elon Musk, Sam 
Altman, and others, who collectively pledged US$1 billion. Musk resigned from the board in 
February 2018 but remained a donor. In 2019, OpenAI LP received a US$1 billion 
investment from Microsoft.” 

https://en.wikipedia.org/wiki/OpenAI 

https://openai.com 

Products include: GPT-3 and DALL-E. 

Founded by Elon Musk, Sam Altmanllya Sutskever, Greg Brockman, Wojciech Zaremba and 
John Schulman. 


Timeline: 11 December 2015-. 


paper is a general term for a student or academic document in general. 
Primarily in PDF when published or handed in. 


In manuscipt/editable/personal form it is generall in the Microsoft Word format. 


PDF ‘Portable Digital Format’. Developed by Adobe. Now free with no license restrictions. 
It is a print to digial medium with few digital affoardances which my work on Visual-Meta 
expands to allow for users to interact with the document in useful ways, while staying 


compatibel with the basic PDF format. 


#Infrastructure. Timeline: Introduced 15 June 1993-. 


Peter Wasilko is an Attorney, Programmer, and Indepedent Scholar. 


He holds a J.D., LL.M., and Certificate in Law, Technology, and Management from Syracuse 
University’s College of Law. He is admitted to practice law in New York State and is a 
member of the New York State Bar Association and its Intellectual Property Section. He also 
maintains memberships in the ACM, IEEE Computer Society, and Association for the 
Advancement of Aritificial Intelligence. His primary technical interests are Hypertext, 
Intelligence Augmentation, Human-Computer Interaction, Computer Supported Cooperative 
Work, Symbolic AI, Law and AI, and Programming Language Design. His intertests beyond 
computing focus on Foresight and Futures Studies with an emphasis on University Futures 
and forays into a wide range of related disciplines ranging from Innovation Studies, Design 
Futures and Speculative Design; Medieval Universities and Trade Guilds; World’s Fairs and 
Theme Parks; TechnoCities and Prototype Communities of Tomorrow; Architecture; and 
Urban Planning. He founded Founders’ Quadrangle — an unincorporated association of 
academics explorng the design space for Universities and Quasi Academic Enterprises of the 


Future. He can be found with email, LinkedIn, ORCID iD, and Twitter. 


#person. Timeline: -. Lives in: New York, New York, USA. 


Pol Baladas is the CEO and Artisan at Fermat, a computational medium where people can 
build their own tools and use tools built by others (we're basically bringing back HyperCard). 
Fermat is made inside Batou.xyz, an industrial research lab that rethinks how people interact 
with tools, computers and ideas. Interested in tools, media, thought and a better future. 
Computing philosopher and charlatan. 

Interested in Tools, thought, interfaces and a better future. 

https://twitter.com/polbaladas 


#person. Timeline: - 


Quest 2 The headset we currently use the most in the Future Text Lab. It sold more than 
Xbox in 2021 and next year we will see the Apple HMD which will enlarge the field of VR/ 
AR even further. 


#Hardware | Timeline: 13 October 2020- | Company: Meta 


Reader a PDF viewer from the Augmented Text company. 


https://www.augmentedtext.info/reader 


Reader Software is a minimalist PDF viewer for macOS. 

Can read any PDF and can provide added interactions if the PDF has Visual-Meta attached. 
This can either be produced by Author or any other word processor with Visual-Meta 
capability, or downloaded from an online repository which features Visual-Meta, such as the 
ACM digital library. Produced by The Augmented Text Company LTD, with programming by 
Jacob Hazelgrove. 

https://www.augmentedtext.info 


#Software. Timeline: 12 July 2019- 


references is a list of all the citations a document uses, in an Appendix. In-Body citation, 
point to these References. This language is not fixed, it is sometimes used interchangeably 
with Bibliography but in my context a Bibliography is a list of work not expressly cited but 
which are relevant. 

in this context ‘Reference’ with uppercase ‘R’ refers to the appendix in an academic 
document which lists cites sources. In contrast, the citation in the body of the document is 
referred to as in-body citation. 

A form of metadata. 


#term 


Relative Dimension A dimension capturing the ordering of data objects based on the 


pairwise application of a comparison function. 


Result Set The set of data object satisfying a database search query. 


Richard Snyder Electronic Literature Lab, Washington State University Vancouver. 


Contributed to this book with Dene Grigar. 


Russia or the Russian Federation, is a transcontinental country spanning Eastern Europe and 


Northern Asia. It is the largest country in the world. 


Sam Brooker I am an academic specialising in digital communication. His research explores 
the relationship between digital technologies & theories of literature and culture. 

Associate Professor of Digital Communications at Richmond American University London, 
UK. 

His research explores the relationship between digital technologies & theories of literature 
and culture. He is a regular contributor to ACM Hypertext and Social Media, frequently 
serving on the programme committee. Additional contributions have featured at the ICIDS 
and SHARP conferences. I am a member of the PRCA International University Advisory 
Group and academic reviewer for numerous journals. 
https://www.richmond.ac.uk/school-of-communications-arts-social-sciences/dr-sam-brooker/ 


#person. Timeline: -. Research interests: Digital cultures. Electronic literature. Book history. 


Digital humanities, Theories of authorship. Transmedia. Lives in: London, UK. 


Scott Rettberg is professor of digital culture in the department of linguistic, literary, and 
aesthetic studies at the University of Bergen, Norway. 

Rettberg was the project leader of ELMCIP (Electronic Literature as a Model of Creativity 
and Innovation in Practice), a €1.000.000, six nation, HERA-funded collaborative research 
project, from 2010-2013. 


Leader of the Bergen Electronic Literature Research Group. Director of the ELMCIP 


Electronic Literature Knowledge Base. Rettberg is the author or coauthor of novel-length 
works of electronic literature, combinatory poetry, and films including The Unknown, Kind 
of Blue, Implementation, Frequency, The Catastrophe Trilogy, Three Rails Live, Toxi*City, 
Hearts and Minds: The Interrogations Project and others. 

His creative work has been exhibited online and at art venues including the Venice Biennale, 
Santa Monica Museum in Barcelona, the Inova Gallery, Rom 8, the Chemical Heritage 
Foundation Museum, Palazzo dell Arti Napoli, Beall Center, the Slought Foundation, The 
Krannert Art Museum, and elsewhere. Cofounder and served as the first executive director of 
the nonprofit Electronic Literature Organization, where he directed major projects funded by 
the Ford Foundation and the Rockefeller Foundation. Rettberg served on the ELO board of 
directors from 2001-2015. Rettberg’s Electronic Literature (Polity, 2018) is the first 
comprehensive study of the history and genres of electronic literature. Rettberg was recently 
awarded a SAMKUL grant from Research Council of Norway to lead a four year research 
project “Extending Digital Narrative.” 

https://twitter.com/scottrettberg 


#person. Timeline: 1970. Born: Norway. Lives in: Bergen, Norway. 


Search Search in this context refers to instant search enabled by Liquid. 


Softspace “Softspace is inventing a new kind of tool for thought for thinkers and makers 


using virtual and augmented reality.” 
A tool for thought. 
https://soft.space 


#Software | Timeline: -. Keywords: VR. AR. 


Spain https://en. wikipedia. org/wiki/Spain 


#location. Timeline: -. 


Spatial Computing “was defined in 2003 by Simon Greenwold, as “human interaction with 
a machine in which the machine retains and manipulates referents to real objects and spaces”. 
With the advent of consumer VR, AR and mixed reality, companies use ‘spatial computing’ in 
reference to the practice of using physical actions (head and body movements, gestures, 
speech) as inputs for interactive digital media systems, with perceived 3D physical space as 


the canvas for video, audio, and haptic outputs. It is also tied to the concept of ‘digital twins'. 


https://en.wikipedia.org/wiki/Spatial_ computing 


SSTV (Slow-Scan Television) is a picture transmission method, used mainly by amateur 


radio operators, to transmit and receive static pictures via radio in monochrome or color. 


https://en.wikipedia.org/wiki/Slow-scan_ television 


stable diffusion is a deep learning, text-to-image model released by startup StabilityAI in 
2022. 

Keywords: collective intelligence. augmented technology. AI. 

#technology. 

https://en.wikipedia.org/wiki/Stable_Diffusion https://stability.ai 


Stephen Fry is an English actor, broadcaster, comedian, director and writer. 
Contributor also to volume | of The Future of Text. 


Timeline: 24 August 1957-. Born: UK 


Stigmergy (/'stigmord3i/ STIG-mor-jee) is a mechanism of indirect coordination, through the 
environment, between agents or actions. 
The principle is that the trace left in the environment by an individual action stimulates the 


performance of a succeeding action by the same or different agent. Agents that respond to 


traces in the environment receive positive fitness benefits, reinforcing the likelihood of these 


behaviors becoming fixed within a population over time. 
https://en.wikipedia. org/wiki/Stigmergy 


#concept 


Ted Nelson, Theodor Holm Nelson is an American pioneer of information technology, 


philosopher, and sociologist. 
He coined the term ‘hypertext’. 
Presented at The Future of Text Symposium. 


#person. Timeline: 17 June 1937-. 


Text is the basic ’stuff’ of this work. 


The Future of Text Annual Symposium and Book Series (first volume published 2020) as 
well as community for fostering dialogue around the future of text which I started over a 


decade ago and which is often co-hosted or presented by Vint Cerf. 


https://futuretextpublishing.com 


The Future of Text Symposium is a series of symposia, books and Journal under the name 


‘The Future of Text’ produced by the same people who run The Augmented Text Company. 


Co-presented by Frode Hegland, Vint Cerf and Ismail Serageldin. Curated by Dene Grigar, 
Claus Atzenbeck and Mark Anderson. 


This is considered as level C of Doug Engelbart’s Three Levels of Activity. 
https://futuretextpublishing.com 


#Event | Timeline: 2011- 


Tom Standage Deputy Editor of The Economist. 
Contributor to The Future of Text volume 1. 


Born: UK. 


tool for thought is both a book and a category of tools, primarily software tools, to augment 
how people think. 

The book: “Tools for Thought: The History and Future of Mind-Expanding Technology is a 
work of "retrospective futurism" in which Smart Mobs author Howard Rheingold looked at 
the history of computing and then attempted to predict what the networked world might look 
like in the mid-1990s. The book covers the groundbreaking work of thinkers like Alan 
Turing, John von Neumann, and J.C.R. Licklider, as well as Xerox PARC, Apple Computer, 
and Microsoft (when Microsoft was "aiming for the hundred-million-dollar category"). 
Rheingold wrote that the impetus behind Tools for Thought was to understand where "mind- 
amplifying technology" was going by understanding where it came from.” https:// 


en.wikipedia.org/wiki/Tools_for_ Thought 


UK https://en.wikipedia.org/wiki/United_Kingdom 


#Location 


Ukraine https://en.wikipedia.org/wiki/Ukraine 


#Location 


University of Southampton University in the UK. 


#Institution. Timeline: 1862-. Southampton, UK 


USA https://en.wikipedia.org/wiki/United_States 


#Location 


ViewSpec Engelbart Concept: View Specifications for a user to see their work in different 


views. 


#dougconcept #term 


Vint Cerf, Vinton Gray Cerf, Vinton G. Cerf is cohost of the Future of Text Symposium. 
Coinventor of the Internet, VP and Chief Internet Evangelist for Google. Chairman of the 
Marconi Society. Former executive at MCI, the Corporation for National Research Initiatives, 
the Internet Corporation for Assigned Names and Numbers (ICANN), the American Registry 
for Internet Numbers (ARIN), the Association for Computing Machinery (ACM) and 
member of the Faculty of Stanford University. Fellow of IEEE, ACM, BCS, AAAS, 
American Academy of Arts and Sciences, and American Philosophical Society. Member of 
the US National Academies of Engineering and Science and foreign member of the Royal 
Society and the Royal Swedish Engineering Society. 


#person. Timeline: 23 June 1943-. 


Visual-Meta An open and robust way to augment flat PDF documents to make them more 


interactive. 


Frode Hegland’s PhD thesis. Realised in the Author software and Reader software by The 
Augmented Text Company. Intended to support Doug Engelbart’s notion of hyperdocuments 


and xfiles. 
Inspired by the BibTeX format and the colophon in books. 
http://visual-meta.info 


#Infrastructure 


Visual-Meta A open and robust way to augment flat PDF documents to make them more 


interactive, by The Augmented Text Company people. 


http://visual-meta.info 


VR VR, ‘Virtual Reality’ is a technology whereby the user is visually immersed in a 
computer generated world. When this world allows the physical world to also appear as 


background, it is refered to as AR ‘Augmented Reality’. 


#Technology | Timeline: 20th century- 


VR VR, ‘Virtual Reality’ is a technology whereby the user is visually immersed in a 
computer generated world. When this world allows the physical world to also appear as 


background, it is called AR ‘Augmented Reality’. 


#Technology | Timeline: 20th century- 


Yiliu Shen-Burke Yiliu Shen-Burke is a designer and coder. 

He is building Softspace, a VR / AR knowledge-mapping tool for creative people (https:// 
www.soft.space). Previously, he was a research resident at Studio Olafur Eliasson in Berlin. 
He has a B.A. in econometrics from Columbia University, and dropped out of the Harvard 
Graduate School of Design. He creates out of a love for making beautiful things that elevate 
those who use them. 

https://twitter.com/yiliu_shenburke https://www. yiliu.sh 

He is featured in issue 1.4 of our Journal: https://futuretextpublishing.com/journal/ 


#Person. Timeline: - Keywords: tool for thought. Spatial computing. 


Endnotes 


 hnttps://youtu.be/FJhXh4eboS8 


* Reader and Author apps are described at: 


https://www.augmentedtext.info 


Download Reader (AppStore): 
https://apps.apple.com/gb/app/reader/id 1179373118 


Download Author (AppStore): 
https://apps.apple.com/gb/app/author-basic/id1587711811 


: https://twitter.com/StrangeNative/status/1562450155080597505? 
https://www.theguardian.com/technology/2017/nov/12/jaron-lanier-book-dawn-new- 
everything-interview-virtual-reality 

5 https://twitter.com/andreeavr/status/1211743140144721922 


https://lisafeldmanbarrett.com/about/ 


a https://www.google.com/url? 
sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUK Ewik3e3 BO8H6AhWK QfEDHZI1V 
CpoQFnoECAcQAQ&url=https%3A%2F%2Fpodcasts.apple.com%2Fus%2Fpodcast%2Flis 


a-feldman-barrett-on-emotions-actions-and-the- 
brain%2Fid1406534739%3Fi%3D 10004988398 19&usg=AOvVaw2nEVnX_ zoNj90NAdyjX 
ZSS 


8 Singular: Zettelkasten (Englisg), Zettelkasten (German) 


Plural: Zettelkastens (Englisg), Zettelkasten (German) 


Wikipedia: 


https://en.wikipedia. org/wiki/Zettelkasten 
? https://royalsocietypublishing.org/doi/10.1098/rstb.2019.0319 
iY https://zettelkasten.de/posts/overview/ 


i Though perhaps that's something we should get comfortable with as society — that people 


are continually evolving as their own ideas and knowledge change. 
12 https://www.oculus.com/desktop/ 


'5 Th the past thirteen years or so since our collection High Wired: On the Design, Use, and 
Theory of Educational MOOs (University of Michigan Press, 1998; 2nd ed. 2001) appeared, 
we have presented at numerous conferences, conducted workshops, and written essays on 
MOOs. We want to thank Markku Eskelinen and Raine Koskimaa for inviting us to publish 
this High Wired ‘redux,’ which folds in an earlier version of the essay by Cynthia Haynes 
(“In Visible Texts: Memory, MOOs, and Momentum’) with talks given by Jan Holmevik and 
Cynthia Haynes at the Computers and Writing Conference in Ann Arbor, Michigan (2011). 
Victor J. Vitanza also presented on that panel; and thus the idea for this collection was 


conceived. 


'4 For an interesting example of vintage newsreel, bouncing ball singalong, and scrolling 
text, see this montage of Jill Sobule’s “Resistance Song”: http://www.youtube.com/watch? 


v=gyUk1tv6CUU. 
'S See The CELL Project, https://cellproject.net. 


16 See this guide for an example: Cripping the Arts. Access Guide. Tangled Arts & Disability, 
January, 2019. http://tangledarts.org/wp-content/uploads/2019/01/cta-access-guide-spreads- 
digital.pdf 

'7 ‘Acton, Kelsie, On Plain Language, Critical Design Lab, Jan 14, 2021 https:// 
www.mapping-access.com/blog-1/2021/1/14/on-plain-language 

Simple English: use common words so readers aren’t stopped by unfamiliar words, use an 
active voice, short sentences. Use headings, lists, bullet points and white space to make 
information clear. And when you do need to use complicated words because they convey a 


big, complicated thing quickly, make sure you define them. 


'8 Th 1985 and 1986, respectively, I published two texts on space art. They are reproduced in: 
E. Kac, Luz & Letra. Ensaios de arte, literatura e comunicacao [Light & Letter. Essays in art, 
literature and communication], Rio de Janeiro, Editora Contra Capa, 2004, pp. 32-34 and pp. 
65-74. Albeit not discussed in the present paper, my creation and development of 
telepresence art since 1986 also engages with the materialities of space, since the bulk of 
space exploration is carried out through telerobotics. For my work with telepresence art, see: 
E. Kac, Telepresence and Bio Art — Networking Humans, Rabbits and Robots, University of 
Michigan Press, Ann Arbor, 2005. An extensive interview about my space art was published 


as follows: J. André, Entretien avec Eduardo Kac, Espace(s) 9 (2013), 131-146. 


'9 The documentary was produced by Observatoire de l'Espace (Space Observatory), the 
cultural lab of the French Space Agency, with assistance from the Daniel and Nina Carasso 
Foundation; it was published by a.p.r.e.s editions and is distributed by Les presses du réel, 


France. 


20 Tn 2019 I produced Adsum (Proof-of-Concept version), which consisted in an edition of 


five 4x4x4 in (10x10x10 cm) laser engraved glass cubes. 


*! T conceived and produced Adsum in 2019, during a Maison Malina Residency in Paris, 
curated by Annick Bureaud and organized by Leonardo/Olats with the support of Fondation 
Daniel & Nina Carasso. Adsum flew in 2022 to the ISS with the support of the Stichting 
Moon Gallery Foundation. Adsum was publicly presented for the first time at the 
EuroMoonMars workshop, November 18-20, 2019, realized at the European Space Research 
and Technology Center (ESTEC), Noordwijk, Netherlands. 


2 The Regex version of Adsum uses only letters to reinterpret the visual symbols that make 
up the work, as stipulated by NASA through a ‘regular expression’ script when the agency 
collected “symbols of cultural significance” for the flight (per NASA’s press release of 
August 5, 2022). As such, my Adsum (Regex) was included in a USB drive that was placed 
aboard Orion for travel around the Moon during Artemis | (as listed in NASA’s Artemis 1 
Official Flight Kit). 


23 such as this, which is hidden until you click on it. 


ot https://youtu.be/PJqbivkm0Ms 

2 https://www.edge.org/conversation/jaron_lanier-one-half-a-manifesto 
2 http://www.jaronlanier.com/agentalien.html 

af https://www.nytimes.com/1997/03/13/style/chronicle-073679.html 
8 Bob Horn’s infographics: 

http://www.bobhorn.us 

>° The Pixel: 

https://en.wikipedia.org/wiki/Pixel 

20 http://1997.webhistory.org/www.lists/www-talk.1993q1/0182.html 
Proposed new tag: IMG [Marc Andreessen] (25 Feb 93): 
http://1997.webhistory.org/www.lists/www-talk.1993q1/0182.html 
Inlined image demo/explanation [Marc Andreessen] (14 Mar 93): 


http://1997.webhistory.org/www.lists/www-talk.1993q1/0263.html 


32 <IMG> and ISMAP [Dave Raggett] (1 Jun 93): http://1997.webhistory.org/www.lists/ 
www-talk.1993q2/0421.html 


Dave Raggett notes: 


“Tim BL reminded me on the phone why this approach can't work in general. Pixels only 
make sense for image data NOT for drawings, equations etc. This means that we really do 
have to consider scaling the mouse click coordinates. Tim suggests we use real numbers with 


0 to 1.0 for each axis (assuming square borders) with the upper left as (0, 0).” 
33 https:/Avww.w3.org/TR/2018/SPSD-html32-201803 15/#map 

a https://www.w3.org/TR/2011/WD-htm15-20110525/the-map-element.html 
= https://en.wikipedia.org/wiki/Adobe_ImageReady 

28 https://en.wikipedia.org/wiki/Raster_ graphics 

al https://en.wikipedia.org/wiki/Vector_ graphics 


38 https://en.wikipedia.org/wiki/Scalable_ Vector Graphics 


5 https://en.wikipedia.org/wiki/Canvas_element 


40 WebGL has an ‘overlay layer’ that can be targeted via its API. However, that is some 
conceptual distance from the simple concept of an image map. It requires noticably more, and 


more complex, code. 

# https://en.wikipedia.org/wiki/Rich_Text_Format 

a2 https://en.wikipedia.org/wiki/Rich_Text_Format_Directory 
4 https://www.w3.org/TR/WD-DOM/introduction.html 

o https://derivative.ca/ 


a8 https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/ 
2023/ENU/Maya-Basics/files/GUID-5EC40DB 1-FBD9-4553-A2FD-6D3508C9B868- 
htm.html?st=hypergraph 


2 https://puredata.info/ 

a https://docs.unrealengine.com/5.0/en-US/blueprints-visual-scripting-in-unreal-engine/ 
ap https://www.khronos.org/opengl/wiki/SPIR-V - accessed October 2022 

oF https://www.haskell.org/ 

70 https://www.tensorflow.org/ 

zz https://www.midjourney.com/home/ 

- https://openai.com/dall-e-2/ 


>3 T'm interested in looking at history to find counterexamples that defamiliarize our ideas of 


what 'text' is and what it can be. Manuscripts, odd printed books, other languages... 


https://twitter.com/rsnous/status/1261296867494653955 

>4 hnttps://twitter.com/rsnous/status/1470009606285914114 
>> https://twitter.com/rsnous/status/1300623059963977730 
>© hnttps://twitter.com/rsnous/status/1464506090146746368 


>” hhttps://twitter.com/rsnous/status/11338630445 12079873 


cal eit repelled by the entire concept of 'figures', which feel like historical baggage inherited 
from the printing press. The idea that you write a bunch of text and then provide a bunch of 
figures, and then it's the job of the typesetter to put those together — it's very strange to me. 
There was a time when authors might leave spelling and punctuation to the editor, too, and 
now that seems absurd — shouldn't 'figures' (graphics) be just as integral a part of the text? 
Shouldn't you, as the author, interleave them as finely with your prose as you possibly can to 
communicate your message? To be honest, I wonder if the way we conceptualize 'text' and 
the future of text is all just an artifact of the printing press, for better or worse. It's contingent; 
it's something people invented and developed; it's not part of nature. (You can imagine other 
cultures and other universes where there is no analogous notion of text.) 'Text' is the stuff that 
was easy to print. 

https://twitter.com/rsnous/status/127532209 1005329408 
https://twitter.com/rsnous/status/1277266878889463808 
https://www.reddit.com/r/mesoamerica/comments/g4norl/comment/folc2wn/? 


utm_source=reddit&utm_medium=web2x&context=3 

>? ‘Inttps://twitter.com/rsnous/status/144989 1630794768388 

69 https://twitter.com/rsnous/status/1453 1249273646 12102 

1 https://twitter.com/rsnous/status/136545 1590098817025 

me https://twitter.com/rsnous/status/1487586970528268293 

63 https://twitter.com/andy_matuschak/status/14897822407499 16165 
oe https://twitter.com/rsnous/status/1327901527072116736 

65 hnttps://twitter.com/rsnous/status/1257801104030482432 


66 When they were making Super Mario 64, the first thing that they did was create a room 
where you just run around as Mario. "We wanted to make a game where just moving Mario 


around was fun." I think about that a lot. 


https://twitter.com/rsnous/status/1257902565850640386 


67 hnttps://twitter.com/rsnous/status/149795439326 1498378 


68 hittps://twitter.com/rsnous/status/1480632170322534406 
” https://twitter.com/rsnous/status/1428779758964248580 
70 https://twitter.com/rsnous/status/1351375798 142267392 
7) hnttps://twitter.com/rsnous/status/1460108841899597826 
?2 Inttps://twitter.com/rsnous/status/1223438844 126613505 
” https://twitter.com/rsnous/status/1180705812433391616 
is https://twitter.com/rsnous/status/1180726548371931136 
ue https://omar.website/posts/against-recognition/ 


a https://acrobatfaq.com/atbref9/index/Windows/DocumentWindow/Viewpane/Mapview/ 


Adornments/Adornmentactions.html 


a https://acrobatfaq.com/atbref9/index/Windows/DocumentWindow/Viewpane/Mapview/ 


Adornments/SmartAdornments.html 
2B http://www.eastgate.com/Tinderbox/ 


fe https://monoskop.org/images/c/c 1/ 
Benedikt Michael _ed Cyberspace First Steps _1991.pdf 
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Visual-Meta Appendix 


This is where your document comes alive. The information in very small type below allows software to 
provide rich interactions with this document. 
See Visual-Meta.info for more information. 


This is what we call Visual-Meta. It is an approach to add information about a document to the document itself on the same level of the content. The same as would be necessary on a physically printed page, as opposed to a data layer, since this data layer can be lost and it makes it 
harder for a user to take advantage of this data, | Important notes are primarily about the encoding of the author information to allow people to cite this document. When listing the names of the authors, they should be in the format ‘last name’, a comma, followed by “first name” 
then ‘middle name’ whilst delimiting discrete authors with (‘and’) between author na_mes, like this: Shakespeare, William and Engelbart, Douglas C. 4 Dates should be ISO 8601 compliant. §] The way reader software looks for Visual-Meta in a PDF is to parse it from the end of the 
document and look for @ {visual-meta-end). If this is found, the software then looks for {@{visual-meta-start} and uses the data found between these marker tags. It is very important to make clear that Visual-Meta is an approach more than a specific format and that it is based on 
wrappers. Anyone can make a custom wrapper for custom metadata and append it by specifying what it contains: For example @dublin-core or @rdfs. § This was written Summer 2021. More information is available from hitps://visual-meta.info or from emailing 
frode@hegland.com for as long as we can maintain these domains. 


@{visual-meta-start} 


@{visual-meta-header-start} 


@visual-meta{ 
version = {1.1}.{generator = {Author 8.6 (1205)}.4)} 


@{visual-meta-header-end} 


@{visual-meta-bibtex-self-citation-start} 


@book {2023-09-24T17:02:24Z/TheFutureo, 
editor = {Frode Alexander Hegland} f{title = {The Future of Text |ll} filename = {FoT 3 (2022).pdf} month = {sep} year = {2022} flinstitution = {Future text Publishing} vm-id = {2023-09-24T17:02:24Z/TheFutureo} 4} 


@ {visual-meta-bibtex-self-citation-end} 


@ {references-start} 


@misc {andyCampbell/DreamingMe, 

author = {Andy Campbell} {title = {Dreaming Methods} {year = {2022} month = {10},} 
@article {andyClarkDavidChalmers/TheExtende, 

author = {Andy Clark and David Chalmers} {title = (The Extended Mind} {journal = {Analysis} [publisher = {[Analysis Committee, Oxford University Press]} year = {1998} pages = {7--19} §pageRange = {7--19} volume = {58},} 
@book {annieMurphyPaul/TheExtende, 

author = {Annie Murphy Paul} title = (The Extended Mind} {publisher = {Eamon Dolan Books} {year = {2021} furl = {http://play. google.com/books/reader?id=Dk-_ DwAAQBAJ} isbn = {9780544947665, 0544947665 },4} 

@misc {davidBClear/Zettelkast, 

author = {David B Clear} {title = { Zettelkasten) year = {2019} $month = {3} url = {https://writingcooperative.com/zettelkasten-how-one-german-scholar-was-so-freakishly-productive-997e4e0ca125},} 

@misc {robertHorn/Mural1, 

author = {Robert Horn} ‘title = {Mural 1}.year = {2022} month = {9}.4]} 

@misc {brandelZachernuk/BobHornMur, 

author = {Brandel Zachernuk} {title = {Bob Horn Mural} {year = {2022} month = (2},§} 


@{references-end} 
@{glossary-start} 


@entry { 

name = {.liquid} {description = {the document format for the Author Software. It is a macOS wrapper with JOSN and RTFD. This is a free and open standard optimised for VR. 

#Infrastructure Timeline: -}.4]} 

@entry{ 

name = {6DoF} description = {Six degrees of freedom (6DOF) refers to the freedom of movement of a rigid body in three-dimensional space. Specifically, the body is free to change position as forward/backward (surge), up/down (heave), lef/right (sway) translation in three 
perpendicular axes, combined with changes in orientation through rotation about three perpendicular axes, often termed yaw (normal axis), pitch (transverse axis), and roll (longitudinal axis). Three degrees of freedom (3DOF), a term often used in the context of virtual reality, refers 
to tracking of rotational motion only: pitch, yaw, and roll, Used by us in VR context. 


See: https://en.wikipedia.org/wiki/Six_degrees_of freedom 
Mh 

@ontry 

name = {academic} {description = {someone who reads academic documents, usually associated with an academic institution, but not necessarily. 

#Profession | Timeline: },4)} 

@entry { 

name = {ACM} description = {From Wikipedia: “The Association for Computing Machinery (ACM) is a US-based international learned society for computing. It was founded in 1947 and is the world’s largest scientific and educational computing society.” 
https://en.wikipedia.org/wiki/Association_for_Computing Machinery 

ACM's ‘Hypertext’ Conference piloted Visual-Meta in 2021. 

#Institution. #Timeline: 1947-. USA}.4} 

@entry { 

name = {Adam Wern} {description = {Working on spatial hypertext software. }.4)} 

@entry { 

name = {Adam} falt-namel = {Adam Wern} {description = {Future Text Lab collborator. } 4} 

@ontry | 

name = {Addressability} description = {refers to how something can refer to something else. In the digital world this primarily means in a local file structure and a server based network. In academia is can refer to citation or references. In the physical world it can refer to a formal 
location layout, such as a flat so and so in house so and so, as well as co-ordinates. It can also man relative addressing, such as saying take a left after the third yellow house. 

#dougeoncept #term}.4} 

@entry { 

name = {Adobe} (description = {“originally called Adobe Systems Incorporated, is an American multinational computer software company incorporated in Delaware(3] and headquartered in San Jose, California. It has historically specialized in software for the creation and 
publication of a wide range of content, including graphics, photography, illustration, animation, multimedia/video, motion pictures, and print.” https://en.wikipedia.org/wiki/Adobe_Inc. 

Founded by: John Warnock and Charles Geschke. 

Hinstitution, #timeline: December 1982.}.)} 

@entry { 

name = {AI} {description = {*Artificial intelligence’ is “intelligence demonstrated by machines, as_ opposed to the natural intelligence displayed by animals including humans.” https://en.wikipedia.org/wiki/Artificial_intelligence 

#Technology. Timeline: 20th century-. Keywords: ML. Machine Learning.}.4)} 

@entry { 

name = {Alan Laidlaw} (description = {https://twitter.com/alanlaidlaw 

person. Timeline: },{} 

@entry { 

name = {Alan Laidlaw} {description 
@ontry 

name = {anaphora} description = {“the use of a word referring back to a word used earlier in a text or conversation, to avoid repetition, for example the pronouns he, she, it, and they and the verb do in I like it and so do they.” 

From Oxford Languges. 

Mh 

@entry { 

name = {Andreea Ion Cojocaru} description = {Andreea Ion Cojocaru is a licensed architect and a software developer. 

She is the co-founder and CEO of NUMENA, an award-winning German company. Andreea works at the intersection of traditional architecture and immersive technologies to develop projects that require a new approach to cognitive and spatial challenges. Currently, she is 
working on a tool that allows users to create, share and edit environments at 1:1 scale. She is also leading projects that mix physical and virtual elements to explore social and economic models based on collective agency across multiple spatial modalities. 

She is featured in issue 1.4 of our Journal: hitps://futuretextpublishing.com/journal/ 

person. Timeline: -. Company: NUMENA. Lives in: Germany. Key interests VR / AR / AL, embodiment, cognitive neuroscience / cognition, architecture, Works with: Bob Horn, Claus Atzenbeck.}.4} 

@entry { 

name = {Andy Campbell} {description = {is the Digital Director for the arts/media organisation One to One Development Trust and the founder/director of Dreaming Methods - an immersive digital storytelling studio. 

‘A judge and speaker at the New Media Writing Prize since its inception in 2010, he has been producing electronic literature, digital art and experimental narrative games for over 25 years and has won many international awards. He is a speaker and workshop leader giving 
masterclasses in Unity and digitally delivered storytelling. Often called upon by international arts and culture organisations as well as universities across the globe, his work includes Lead Developer for the multi-award winning episodic digital novel Inanimate Alice and services to 
the Orient Foundation for Arts and Culture creating the world’s largest online archive of digitized Tibetan cultural resources. His most recent collaborative work Monoliths with Pilot Theatre and One to One Development Trust has been nominated for the Immersive Art/XR award 
at the 2022 BFI London Film Festival. 

https://twitter.com/dreamingmethods facebook.com/dreamingmethods https://\www instagram.conv/dreamingmethods/ 

person. Timeline: 23 April 1975-. Born in: Halifax, UK. Company: One to One Development Trust and Dreaming Methods www.onetoonedevelopment.org www.dreamingmethods.com. Currently: Wakefield, UK. Interests: Digital fiction, digital art, VR, WebGL, HTMLS, video 
games, game engines, 3D graphics, software, programming, digital environments, immersive audio, Al, XR, TV, electronic music, film, cinema, books, experimental literature, poetry, literacy, engagement, accessibility, inclusivity, collaboration, education, web design, CSS},4} 
@ontry { 

name = {Anne-Laure Le Cunff} {description = {Founder of Ness Labs and contributor to The Future of Text volume 1.}.4)} 

@entry { 

name = {Annie Murphy Paul} {description = {From an interview in GQ: “We make better use of our cognitive resources, says Paul, when we use them in conjunction with “extra-neural” resources: our body (embodied cognition), our environment (situated cognition), and the 
people around us (distributed cognition). The brain evolved to move the body, to navigate through space, to interact with other people,” says Paul. 


{https://twitter.com/alanlaidlaw},9} 


“Those are these human strengths that we're totally putting aside when we focus on the brain and we think, ‘To get real thinking and real work done, I have to sit still, not talk to anybody, and just push my brain harder and harder.” It's just not working very well.” 
https://www.gq.com/story/extended-mind-annie-murphy-paul 

Author of The Extended Mind. 

person. Timeline: -.},4} 

@entry { 

name = {Apple HMD} description = {an as yet not released VR/AR headset which is expected to be announced late 2022, early 2023 and which will enlarge the market considerably. 

#Hardware | Timeline: 2023 | Company: Apple}.4]} 

@entry { 

name = {Apurva Chitnis} description = {Head Of Engineering, koodos labs. 

Hperson}.4)} 

@entry { 

name = {AR} {description = {Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including 
visual, auditory, haptic, somatosensory and olfactory.” 

hitps://en.wikipedia.org/wiki/Augmented reality 


AR includes the real reality, as opposed to VR which does not.}.4} 
@entry { 

name = {AR} {description = {includes the real reality, as opposed to VR which does not. 

“Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, 
somatosensory and olfactory. 
https://en.wikipedia.org/wiki/Augmented_reality 

#Technology | Timeline: 20th century-}.4)} 

@entry { 

name = {ARC} description = {Engelbart Concept: Augmentation Research Center, The name of Doug's lab at SRI where he proposed a s 
research, later renamed Augment and from which I decided on the name Author, since Author and Augment share etymological roots. 
HInstitution. Timeline: 1960s-1970s. USA}.4)} 

@entry { 

name = {Architectural Spaces} ,{descriptio 
@entry { 

name = {Art} {description = {a category used here to help the reader identify artist and discussions around artworks. } 

@entry { 

name = {Augmented Text Company} {description = {The company which produces the Author software, Reader software and Liquid software. 

Director is Frode Hegland. 

hitps://www.augmentedtext.info 

#Institution. Timeline: 9 March 2016-}.4)} 

@entry { 

name = {Author} {description = {means in the context of this work someone who writes a document, which includes academic papers and books. Can also refer to the Author software by The Augmented Text Company. 

Etymology: “mid-14c., auctor, autour, autor “father, creator, one who brings about, one who makes or creates” someone or something, from Old French auctor, acteor “author, originator, creator, instigator” (12c., Modern French auteur) and directly from Latin auctor “promoter, 
producer, father, progenitor; builder, founder; trustworthy writer, authority; historian; performer, doer; responsible person, teacher,” literally “one who causes to grow,” agent noun from auctus, past participle of augere “to increase,” from PIE root *aug- (1) “to increase.” From 
hitps://www.etymonline.com/search?q~Author} } 

@entry { 

name = {Barbara Tversky} {description = {Guest presenter at the Future Text Lab. 

Issue 1.1 on the 21st of January 2022. 

Professor emerita of psychology at Stanford University and a professor of psychology and education at Teachers College, Columbia University. Tversky specialises in cognitive psychology. 

‘Author of Mind in Motion: How Action Shapes Thought. Basic Books, 2019. 

tiperson}.4)} 

@entry { 

name = {Benediktine Cyberspace} {description = {A 3-D visualization inspired by Michael Benedikt’s seminal text, “Cyberspace: First Steps”}.]} 

@entry { 

name = {BibTeX} {description = {is a specific format for conveying citation information within the LaTex environment, developed by Oren Patashnik and Leslie Lamport, released in 1985. The benefit of the system was to separate citation information from presentation style and it 
is human readable, though it slightly looks like code. It inspired the format of Visual-Meta and Visual-Meta contains a straight BibTeX section to allow the document which contains it to be cited.},} 

@entry { 

name = {Bob Horn} falt-namel = {Robert E. Horn} {description 
producing large scale murals. 

Was Senior Researcher at Stanford University's Human Science and Technology Advanced Research Institute (H-STAR) for 27 years. While he was a research associate at Columbia University, he created a widely used methodology for the analysis of any complex stable subject 
matter. This research became an international consulting company, Information Mapping, Inc. He was CEO and chairman for 20 years. He has taught at Harvard, Columbia, and Sheffield (UK) universities. Recently, he has been developing large info-murals and leading “mess 
mapping” projects and workshops to enable decision making groups get their minds around larger contexts for strategic discussions of wicked problems and mega-messes. The projects range from global climate change, energy security, nuclear waste disposal, NASA’s research 
programs. He was the synthesizer and visualizer for the World Business Council for Sustainable Development's Vision 2050 project the European-Commission-supported project on policy options for a resource efficient Europe (POLFREE). His development of visual 
argumentation mapping has resulted in the publication of the Mapping Great Debates series, which, in the same year received a full-page review in Nature, as well as being hung as artwork in a national museum in The Hague as part of an exhibit on information design as a fine art. 
His 7 books include Visual Language: Global Communication for the 21st Century (1998) and Mapping Hypertext (1989, a book about the web 2 years before the web was created). His most recent book (still in draft) is: The Little Book of Wicked Problems and Social Messes. 
He has received Lifetime Achievement Awards from the Association of Computing Machinery (ACM) and the International Society for Performance and Instruction. He is a fellow of the World Academy of Art and Science. 

www.bobhorn.us 

person. Timeline: - Lives in: California, USA},} 

@entry { 

name = {Bob Horn} description = {Robert E. “Bob” Horn is a fellow of the World Academy of Art and Science. For 27 years, he was a Senior Researcher the Human Science and Technology Advanced Research Institute (H-STAR), Stanford University, where he worked with 
international task forces and governments on wicked problems an social messes. He has taught at Harvard and Columbia universities and is the author/editor of ten books. 

He produces large scale murals. 

www.bobhorn.us},4} 

@entry { 

name = {Bob Stein} {description = {is the director of the Institute for the Future of the Book 

He founded The Voyager Company in 1985, the first commercial multimedia CD-ROM publisher, and The Criterion Collection in 1984, a collection of definitive films on digital media with in-depth background information (including the first films with recorded audio 
commentary). Worked with Alan Kay at the Atari Research Group on various electronic publishing projects. Currently works on The Tapestry Project. 

person. Timeline: 20 April 1946-},4} 

@entry { 

name = {Book} {description = {a published work, on any substrate, inclduing paper or digital. 


stem called H-LAM/T in 1962 and developed and in 1968 demonstrated NLS: oNLine System, his platform for shared knowledge work 


= {VR Represenations of landscapes, buildings, and interiors designs modeled after the physical world.},} 


{is a political scientist with a special interest in policy communication, social and organizational learning, and knowledge management (especially in sustainability and national security affairs), known for 


Generally longer than a ‘paper’ but in terms of digital work, there is little technical difference, the difference is mostly social.},4} 
@entry { 

name = {Brandel Zachernuk} {description = {AR / VR Creative technologist by day, digitally-embodied cognition enthusiast by night! 

Something that I've been very passionate about trying to investigate and play with over the last 10 years or so is what is the most pedestrian thing that you can do with virtual reality and a technology that was was word processing was reading and reading and thinking about what 
are the basic building blocks of that process of writing and reading that can fundamentally be changed by buying virtual reality? Realising that if you don’t have a screen, you have the ability for information to mean what it means for your purposes, rather than the technical 
limitations that apply as a consequence of a mouse or keyboard or things like that, and also deeply invested in understanding some of the emerging cognitive science and and neurophysiological sort of views about what it is that the mind is and the way that we work best. So reading 
about learning about what people call for in cognition, it's embedded, embodied in active and extended mind and how that might pertain to what we should be doing with software and systems, both as well as hardware if necessary to make it so that we can. We can think properly 
and express properly and stuff. 
https://youtu.be/2NeSCOrVw2: 
hitps://twitter.com/zachernuk 
https://www. youtube.com/channel/UCOP9cT-IwB6oxAcfvS1x_ZQ 
://futuretextlab. info/tag/bybrandel/ 
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#Person. Timeline: -. Lives in: California, USA.},} 
@entry { 

name = {Brett Jackson} {description = {VR developer. 

Currently working on Idea Engine (https://linktr.ee/ideaengine), an interactive mind map. Creator of Dimensional, Jigsaw360, Breath Tech. 

https://twitter.com/JumbliVR} 4} 

@entry { 

name = {Caitlin Fisher} description = {is professor and Chair, Cinema and Media Arts and Director, Augmented Reality Lab, Director, Immersive Storytelling Lab, School of the Arts, Media, Performance and Design, York University 

President, Electronic Literature Organization 

Works with AR and VR and electronic literature, future cinema, emerging technology. 

#person. Timeline: -},9} 

@entry { 

name = {Canada} {description = {https://en.wikipedia.org/wiki/Canada 

#Location}.4} 

@entry { 

name = {Chalktalk} {description = {Chalktalk is a digital presentation and communication language in development at New York University's Future Reality Lab. Using a blackboard-like interface, it allows a presenter to create and interact with animated digital sketches in order to 
demonstrate ideas and concepts in the context of a live presentation or conversation. } 4} 

@entry { 

name = {Charlie Hargood} {description = {Academic in Games Technology at Bournemouth University.} 4} 

@entry { 

name = {Christopher Gutteridge}, {description = {is also a contributor to the first volume of The Future of Text. 

Lives: Southampton, UK. 

Works at the University of Southampton.}.4)} 

@entry { 

name = {citation} {description = {In this work, the term’ citation’ is both the ‘citation’ mention in the body of the document, and its corresponding ‘references’ entry for the source material in the Reference Appendix at the end of the document, though for the sake of clarity I will 
mainly use ‘citation’ to refer to the in-body half and ‘reference’ for the entry of the source in the References section. 


#term 

Mh 

@entry { 

name = {Claus Atzenbeck} {description = {is a cocurator of The Future of Text Symposium. 

He holds a professor position at Hof University, Germany and is leading the Visual Analytics research group at the university’s Institute of Information Systems. 

He served as general co-chair of the 30th Anniversary ACM Hypertext Conference 2019 in Hof, Germany (https://human.iisys.de/ht2019/, 7 @ACMHT), is co-organizer of the HUMAN workshop series (https://human.iisys.de/human/, 7 @HUMAN_HT), and the initiator of the 
Historic Hypertext Project (https://human.iisys.de/hist_ HT/, 7 @hist_HT). 

http://www.atzenbeck.de https://twitter.com/clausatz https://www.iisys.de 

#person. Timeline: . Born: Germany. Lives in: Germany, Research Interests: Hypermedia including spatial and navigational structures, spatial and temporal parsing for spatial hypertexts, and hypertext narratives. A specific focus of his work is on intelligent user interfaces for visual 
analytics.} 4} 


@entry { 

name = {Co-Evolution} 4idescription = {Engelbart Concept: Most capabilities are improved, or augmented, by many interdependent technical and non-technical elements, of which tools make up only a small part: On one hand, there is the human system, which includes paradigms, 
organizations, procedures, customs, methods, language, attitudes, skills, knowledge, training and so on- all of which all exists within the basic perceptual and motor capabilities of the human being. On the other hand, there is the tool system, which includes media, computers, 
communications systems ete. Together, they comprise the augmentation system. 

#dougconcept #term} 4} 

@entry { 

name = {Cognition} {description = {“is the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses”. 

https://en.wikipedia.org/wiki/Cognition}.} 

@entry { 

name = {colophon} description = {“In publishing, a colophon (/'kolafan, -fon/)[1] is a brief statement containing information about the publication of a book such as the place of publication, the publisher, and the date of publication. 
https://en.wikipedia.org/wiki/Colophon_(publishing) 

Orignally, in Mesopotamia, colophons contained quite a bit more information: 

https://colophons-and-scholars.com/home } §j} 

@entry { 

name = {colophon} description = {A publisher's emblem or imprint, usually on the title page of a book which inspired Visual-Meta, which is similar data (in BibTeX form), at the back of a document. 

“In publishing, a colophon (/'kolafan, -fon/)[1] is a brief statement containing information about the publication of a book such as the place of publication, the publisher, and the date of publication.” 
https://en.wikipedia.org/wiki/Colophon_(publishing) 

Originally, in Mesopotamia, colophons contained quite a bit more information: 

https://colophons-and-scholars.com/home 

A form of metadata. }} 

@entry { 

name = {computational text} description = {Vint Cerf likes to use the term ‘computational text’ to mean text which can be interacted with computationally. 

Relates to encoding in Visual-Meta. },{}} 

@entry { 

name = {concept} falt-namel = {defined concept} falt-name2 = {defined concepts} {description = {is, in the context of this book, any text which is defined by an author. 

When using the Augmented Text Tools Author this is exported in a PDF as a Glossary which becomes interactive when opened in Reader. In Author, The Map view uses the definition to draw lines where text from a definition is also present on the Map. When a document is 
exported to PDF the Defined Concepts become Glossary Terms. 

This is as opposed to inferred concept. 

Also aView in Author and in Reader (with Visual-Meta) to show all named entities (and headings). 

#term}, 4} 

@entry { 

name = {CRS} {description = {(Commercial Resupply Services) are a series of flights awarded by NASA for the delivery of cargo and supplies to the International Space Station (ISS) on commercially operated spacecraft. 
https://en.wikipedia.org/wiki/Commercial_Resupply_Services} 4} 

@entry { 

name = {Cyberspace} {idescription = {“The term "cyberspace" first appeared in the visual arts in the late 1960s, when Danish artist Susanne Ussing (1940-1998) and her partner architect Carsten Hoff (b. 1934) constituted themselves as Atelier Cyberspace.” Used in this context to 
mean a digital world, not necessarily in VR or AR.}.4)} 

@entry { 

name = {Cynthia Haynes} {description = {is former Director of Rhetorics, Communication, and Information Design Ph.D program and currently Professor of English at Clemson University. 

Her research interests are rhetoric, composition, multimodal pedagogy, virtual worlds, VR, critical theory, computer games studies, and the rhetoric of war and terrorism. Her recent book, The Homesick Phone Book: Addressing Rhetoric in the Age of Perpetual Conflict (Southern 
Illinois University Press, 2016) won the 2017 Rhetoric Society of America annual book prize. She is currently working on a book, Unalienable Rites: The Architecture of Mass Rhetoric. 

She is, along with Jan Holmevik, member of the Kairos Editorial Board and co-founders of the journal's electronic partner LinguaMOO. 

#person. Timeline: -. Lives in: Clemson, South Carolina, USA.}.4)} 

@entry { 

name = {DALL-E}. {description = {DALL-E (stylized as DALL-E) and DALL-E 2 are machine learning models developed by OpenAl to generate digital images from natural language descriptions. 

The name is a portmanteau of the names of animated robot Pixar character WALL-E and the Spanish surrealist artist Salvador Dali. 

https://en.wikipedia.org/wiki/DALL-E https://openai.com/blog/dall-e/ 

#technology. Al. Machine Learning. ML.}.4)} 

@entry { 

name = {Dame Wendy Hall} falt-namel = {Wendy Hall} {description = {Frode Hegland’s PhD primary supervisor, along with Les Carr and David Millard at the University of Southampton. 
@entry { 

name = {Daniel Dennett} {description = {is an American philosopher, writer, and cognitive scientist whose research centers on the philosophy of mind, philosophy of science, and philosophy of biology, particularly as those fields relate to evolutionary biology and cognitive 
science. 

https://en.wikipedia.org/wiki/Daniel_Dennett 

#person Timeline: 28 March 1942-},} 

@entry { 

name = {David Jay Bolter} {description = {Wesley Chair in New Media, Georgia Tech and author of The Digital Plenitude: The Decline of Elite Culture and the Rise of New Media. 

Along with John B. Smith and Michael Joyce, Bolter co-created Storyspace, a software program for creating, editing, and displaying hypertext fiction. 

Contributor to The Future of Text Volume 1. 

Timeline: 17 August 1951-. Born: USA},} 

@entry { 

name = {David Millard} {description = {Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and Les Carr at the University of Southampton. }.]} 

@entry { 

name = {Deena Larsen}, {description = {is a new media and hypertext author involved in the creative electronic writing community since the 1980s, From USA. 

https://en.wikipedia.org/wiki/Deena_Larsen 

#person. Timeline: 1964-},4)} 

@entry { 

name = {Definition} {description = {when it comes to glossaries, it is the user’s stated meaning of a term.} 4} 

@entry { 

name = {Dene Grigar} {description = {is a cocurator of The Future of Text Symposium, 

PAD, Director of the Electronic Literature Lab; Managing Director and Curator of ELO’s The NEXT. 

Digital artist and scholar based in Vancouver, Washington. Professor and Director of the Creative Media & Digital Culture Program at Washington State University Vancouver, USA. A prioneer of digital literature. Former president of the Electronic Literature Organisation, 
#person. Timeline: -. Electronic Literature. }.]} 

@entry { 

name = {Distance-Independent Millimeters} {alt-namel = {DMM} {description = {Google researchers needed a way to design screens that could be easily read at any distance. To help solve this, they came up with "Distance-Independent Millimeters" or DMMs. A VR and AR 
term. 

More detail at: https://www. youtube.com/watch?v=ES9jAtHRFHQ 

#term 

Mh 

@entry { 

name = {DOI} description = {*Document Object Identifiers’ 

An effort to make addressing academic documents via the web more robust. 

Used in Author software to let the user paste a DOI to cite an academic document which is then sent to CrossRef to be parsed into BibTeX which is then used to create a full citation. 

#Infrastrueture} } 

@entry { 

name = {Doug Engelbart} falt-namel = {Douglas Carl Engelbart} jalt-name2 = {Doug} description = {From Wikipedia: “He was an engineer and inventor, and an early computer and Internet pioneer. He is best known for his work on founding the field of human-computer 
interaction, particularly while at his Augmentation Research Center Lab in SRI International, which resulted in creation of the computer mouse, and the development of hypertext, networked computers, and precursors to graphical user interfaces. These were demonstrated at The 
Mother of All Demos in 1968, Engelbart’s law, the observation that the intrinsic rate of human performance is exponential, is named after him.” 

His seminal paper was ‘Augmenting Human Intellect’, 1962. 

He was also my mentor and greatly influened my work, resulting in my company called The Augmented Text Compant and my word processor being called Author, in honour of his ‘Augment’ system, Visual-Meta is inspired by his Open Hyperdocument work. 

#person. Timeline: 30 January 1925- 2 July 2013}.4} 

@entry { 

name = {Doug Engelbart} description = {“He was an engineer and inventor, and an early computer and Internet pioneer. He is best known for his work on founding the field of human-computer interaction, particularly while at his Augmentation Research Center Lab in SRI 
International, which resulted in creation of the computer mouse, and the development of hypertext, networked computers, and precursors to graphical user interfaces. These were demonstrated at The Mother of All Demos in 1968. Engelbart's law, the observation that the intrinsic 
rate of human performance is exponential, is named after him.” 

https://en.wikipedia.org/wiki/Douglas_Engelbart 

He was also my mentor and greatly influened my work, resulting in my company called The Augmented Text Compant and my word processor being called Author, in honour of his ‘Augment’ system. Visual-Meta is inspired by his Open Hyperdocument work. } §j} 

@entry { 

name = {Eduardo Kac} {description = {is internationally recognized for his groundbreaking work in contemporary art and poetry. 

In the early 1980s, Kac created digital, holographic and online works that anticipated the global culture we live in today, composed of ever-changing information in constant flux. In 1997 the artist coined the term "Bio Art," igniting the development of this new art form with works 
such as his transgenic rabbit GFP Bunny (2000) and Natural History of the Enigma (2009), which earned him the Golden Nica, the most prestigious award in the field of media art. GFP Bunny has become a global phenomenon, having been appropriated by major popular culture 
franchises such as Sherlock, Big Bang Theory and Simpsons, and by writers such as Margaret Atwood and Michael Crichton. In 2017, Kac created Inner Telescope, a work conceived for and realized in outer space with the cooperation of French astronaut Thomas Pesquet. Kac’s 
singular and highly influential career spans poetry, performance, drawing, printmaking, photography, artist's books, early digital and online works, holography, telepresence, bio art, and space art. Kac has also authored or edited several books, including Telepresence and Bio Art 
Networking Humans, Rabbits and Robots (University of Michigan Press, 2005). Kac’s work has been exhibited internationally at venues such as New Museum, New York; Pompidou Center, Paris; MAXXI-Museum of XXI Century Arts, Rome; Mori Art Museum, Tokyo; Reina 
Sofia Museum, Madrid; Power Station of Art, Shanghai; and Seoul Museum of Art, Korea. Kae's work has been showcased in biennials such as Venice Biennale, Italy; Yokohama Triennial, Japan; Gwangju Biennale, Korea; Bienal de Sao Paulo, Brazil; and Bienal de Habana, Cuba. 
His works are in major collections such as Museum of Modern Art-MoMA, New York; Tate Modern, London; Victoria & Albert Museum, London; Museum Les Abattoirs—Frae Occitanie Toulouse, France; Valencian Institute of Modern Art-IVAM, Spain; Museum ZKM, 
Karlsruhe, Germany; and Museum of Contemporary Art of Sdo Paulo, among others. Kac was elected as full member to the LAF (International Astronautical Federation) Technical Activities Committee for the Cultural Utilisation of Space (ITACCUS). 

#person. Timeline: -. 1962 -. Born: Brazil. Lives in: USA .}.4]} 

@entry { 

name = {Egypt} description = {https://en.wikipedia.org/wiki/Egypt 

#Location},§} 

@entry { 

name = {Electronic Literature} description = {“Electronic literature or digital literature is a genre of literature encompassing works created exclusively on and for digital devices, such as computers, tablets, and mobile phones. A work of electronic literature can be defined as "a 


construction whose literary aesthetics emerge from computation", "work that could only exist in the space for which it was developed/written/coded—the digital space”.” https://en. wikipedia.org/wiki/Electronic_literature}.]} 


@entry { 

name = {ESA} {description = {(European Space Agency) is an intergovernmental organisation of 22 member states dedicated to the exploration of space. 
https://en.wikipedia.org/wiki/European_Space_Agency},‘)} 

@entry { 

name = {Extrinsic}.f{description = {Attributes of an object that are represented visually though that object’s spatial position in a coordinate system represening the domain of their possible values. },§} 
@entry { 

name = {Fabien Benetou} {description = {Prototyping - European Parliament Innovation lab WebXR consultant - Former UNICEF Innovation Fund WebXR technical advisor. VR prototypist. 
https://twitter.com/utopiah https://fabien.benetou.fr 

#person. Timeline: 6 November 1982-} 4} 

@entry { 


interest 


name = {Fermat} {description = {is a spatial canvas where every element is programmable by the end user. Users can create their own tools and share them with every other user via a public toolbox. 
Recently, we discovered that the most transformative tools built on top of Fermat were using Artificial Intelligence. 

(We're bringing back HyperCard!). A tool for thought. 

https://fermat.ws 

#Software. Timeline: -. Keywords: Al. GPT-3. Stable Diffusion. } 4} 

@entry { 

name = {flatland} {description = {a semi-humorous term we use to refer to traditional displays, as opposed to the augmented environments of VR and AR. We feel that it is important to be able to move data, including metadata between these environments.},4}} 
@entry { 

name = {Folding} {description = {Folding of a document into a table of contents is enabled through Visual-Meta. } } 

@entry { 

name = {Frode Hegland},falt-namel = {Frode Alexander Hegland} {description = {is cohost of the Future of Text Symposium. 

Director of The Augmented Text Company where he designed the macOS Author word processor, Reader PDF viewer and the Liquid software. https://www.augmentedtext.info/ 

Editor of the 'The Future of Text’ series of books and The Future of Text Journal. 

Designed Visual-Meta. 

His mentor was Doug Engelbart. Greatly influenced by his friend Ted Nelson. 

PhD from the University of Southampton. Advisors Wendy Hall, Les Carr and David Millard. 

#person. #Timeline: Born 2 June 1968-. Born in: Bergen, Norway. Lives in: London, UK.},4} 

@entry { 

name = {Future Reality Lab} {description = {New York University's Future Reality Lab. 


https://frl.nyuedu 

Mh 

@entry { 

name = {Future Text Lab} {description = {a group of people who meet to work on the future of text. Active members currently include Vint Cerf, Adam Wern, Fabien Benetou, Bob Horn, Alan Laidlaw, Mark Anderson, Peter Wasilko and Frode Hegland.}.4)} 
@entry { 

name = {Future Text Lab} {description = {A lab dedicated to studying how working with knowledge, particularly text, can be done effectively in VR/AR/Metaverse. 

We host Guest presenters and publish a Journal which is part of The Future of Text series. 

https://futuretextlab.info 

#Institution. Timeline: - 2021)},4)} 

@entry { 

name = {Gavin Menichini} {description = {Guest presenter at the Future Text Lab. Issue 1.2 on the 25th of February 2022. 

Strategic Account Executive at the VR company Immersed. “Work Faster in VR” 

https://immersed.com},{} 

@entry { 

name = {Gems} {description = {software developed by Lorenzo Bernaschina. 

https://gemsnotes.app/}.4)} 

@entry { 

name = {George Lakoff} {description = {is an American cognitive linguist and philosopher, best known for his thesis that people's lives are significantly influenced by the conceptual metaphors they use to explain complex phenomena. 
https://en.wikipedia.org/wiki/George_Lakoff 

#person Timeline: 24 May 1941 

Mh 

@entry { 

name = {Germany} {description = {https://en.wikipedia.org/wiki/Germany 

#Location} 4} 

@entry { 

name = {Glossary} {description = {means, in the context of this work, a specific user or editor list of definitions for a specific document. 

Defined Concepts in the Author Software is exported as a Glossary. 

This is different from a dictionary since dictionary definitions have general validity. 

Inspired by discussions with Doug Engelbart.},4} 

@entry { 

name = {Glossary} {description = {In the context of my work and thinking, a glossary is a set of Defined Concepts which an author has created and which is then exported as a Glossary for the reader. 

The primary purpose of this is for the author to have to think through their writing by explicitly stating what something is. The definition of the defined concept can then include text which also has a definition and in the Map view in Author this connection can be shown as a line 
when either defined term is selected. 

The secondary purpose is for the author to edit this Glossary to make sure it is coherent for export and it can then help a reader understand the author’s intentions, 

#term}, 4} 

@entry { 

name = {Glossing} {description = {A way of elucidating parts of text. 

https://www.etymonline.com/search?q=gloss},{]} 

@entry { 
name = {Google} (description = 
en.wikipedia.org/wiki/Google 
#Institution. Timeline: 1998-. USA}4} 

@entry { 

name = {GPT} {description = {Generative Pre-trained Transformer” inelduinig GPT-2 and GPT-3. 
https://en.wikipedia.org/wiki/GPT-2 

https://en.wikipedia.org/wiki/GPT-3 

Product of OpenAl.}.4)} 


@entry { 

name = {GPT-3} {description = {Generative Pre-trained Transformer 3° (GPT-3; stylized GPT-3) is an autoregressive language model that uses deep learning to produce human-like text. https://en,wikipedia.org/wiki/GPT-3 
“In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to generate computer code, poetry, and prose is not just "amazing", "spooky", and "humbling", but also "more than a little terrifying”. https://www.worldeat.org/issn/0362-433 1 
Timeline: 11 June 2020-. Keywords: Deep Learning. Al. ML. Machine Learning. } {} 

@entry { 

name = {HMD} {description = {Head Mounted Display for use in VR or AR. 

#Hardware | Timeline: },)} 

@entry { 

name = {HTML} {description = (“The HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser.” https://en.wikipedia.org/wiki/HTML 

A technology we are also using, along with PDF.}.4)} 

@entry { 

name = {hypertext} {description = {a term invented by Ted Nelson for interactive and connected digital text. }{} 

@entry { 

name = {Immersed} {description = {Immersed is a virtual reality product, working productivity software for virtual offices. 

https://immersed.com} } 

@entry { 

name = {Intrinsic} {description = {Attribute of a data object that are not represented by that object’s spatial position in a given visualization but which may be represented by its size, shape, color, transparence, or other positionally independent visualization technique. },4} 
@entry { 

name = {Ismail Serageldin} description = {is a long time collaborator of the Future of Text Symposium, 

Founding Director of the Bibliotheca Alexandrina (BA) and was Vice President of the World Bank. 

#person. Timeline: 1944-. Born: Egypt.}.q)} 

@entry { 

name = {ISS},{description = {(International Space Station) is the largest modular space station currently in low Earth orbit. 

https://en.wikipedia.org/wiki/International_Space_Station},{} 

@entry { 

name = {Jack Kausch) description = {is an academic whose research interests are in linguistics, ontologies and the semantic web, embeddings and the history of writing. 

He is a PhD student at the Western University of Ontario in Canada. 

His advisor is Kamran Sedig. 

He works as a data labeller for OpenAl and is interested in Al. 

#person. Timeline: 14 October 1993-. Born in: Ann Arbor, Michigan. Lives in: Ontario, Canada.} 4} 

@entry { 

name = {Jacob Hazelgrove} description = {is the programmer for the Augmented Text Company for Frode Hegland, including Author and Reader, as well as imlementor of Visual-Meta export from Author and import and interaction in Reader. 
#Person | Timeline: -}.4)} 

@entry { 

name = {Jad Esber} description = {Guest presenter at the Future Text Lab. 

CEO of koodos and an Affiliate at the Berkman Klein Centre for Internet & Society at Harvard. 

https://twitter.com/Jad_AE} } 

@entry { 

name = {Jan Holmevik} falt-namel = (Jan Rune Holmevik} {description = {is Associate Professor of English and Special Advisor to the Vice President for Information Technology & CIO, as well as Past President of the Faculty Senate at Clemson University. 
PhD in Humanistic Informatics from the University of Bergen, Norway and a Master’s degree in the history of technology from the Norwegian University of Science and Technology in Trondheim, Norway. 

He specializes in Interactive and Social Media and conducts research in game design, game culture, digital literacy, social media, visual communication, humanistic informatics, information design, data visualization, social media forensics, transmedia, and electracy. 
http://t.co/70V4voInkm https://twitter.com/holmevik 

#person. Timeline: -},9} 

@entry { 

name = {Jaron Lanier} {description = {“American computer scientist, visual artist, computer philosophy writer, technologist, futurist, and composer of contemporary classical music. 

Considered a founder of the field of virtual reality.” https://en.wikipedia.org/wiki/Jaron_Lanier 

VR pioneer. 

#Person. Timeline: 3 May 1960- . Born: NY USA}4} 

@entry { 

name = {Jim Strahorn} description = {Contributor to the Future Text Lab. }.4]} 

@entry { 

name = {Johannah Rodgers} {description = {Author, “Engineering Language: Teaching Machines to Read and Write in the U.S. 1869 - 1969, 

Contributor to The Future of Text volume 1.},4]} 

@entry { 

name = {Jonathan Finn} {description = {is a developer of creativity apps (e.g. Sibelius music writing system). 


is an American multinational technology company that focuses on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics.” https:// 


Contributor to The Future of Text volume 2 and editing for volume 3. 

timeline: -}.4)} 

@entry { 

name = {Jorge Luis Borges} {description = {“was an Argentinian writer, essayist and translator known for his trademark themes: dreams, labyrinths, libraries, language and mythology. His stories, non-linear narratives that mix fact, fantasy, hox and forgery, are generally considered 
to have reinvented modern literature. This dialogue speculates on Borges’s position regarding language and virtual reality based on his short stories Tlén, Ugbar, Orbis Tertius and Funes, the Memorious. Moreover, the entire conversation makes use of many of Borges’s literary 
techniques. Most of the time I stay close to what the main characters could have plausibly said in such a situation, but, like Borges in his own stories, I also diverge from that and use the two characters to purse my own arguments. Hinted at by the fact that the footage was recorded 
in Borges’s headset, this is the kind of thing he would write." 

https://en.wikipedia.org/wiki/Jorge_Luis Borges 

Timeline: 1899 ~ 1986. Born Argentina.} } 

@entry { 

name = {Kalev Leetaru} {description = {is an American internet entrepreneur, academic, and senior fellow at the George Washington University School of Engineering and Applied Science Center for Cyber & Homeland Security in Washington, D.C. USA. 

Best known for his role as the co-creator of the Global Database of Events, Language, and Tone (GDELT) with Philip Schrodt: "an initiative to construct a catalog of human societal-scale behavior and beliefs across all countries of the world, connecting every person, organization, 
location, count, theme, news source, and event across the planet into a single massive network that captures what's happening around the world, what its context is and who's involved, and how the world is feeling about it, every single day.” 
https://en.wikipedia.org/wiki/GDELT_Project https://blog, gdeltproject.org/web-summit-2017-video! 

#person. Timeline: -.}.4)} 

@entry { 

name = {Ken Perlin} alt-namel = {Kenneth H. Perlin} {description = {is a professor in the Department of Computer Science at New York University, founding director of the Media Research Lab at NYU, director of the Future Reality Lab at NYU, and the Director of the Games 
for Learning Institute. 

B.A. degree in Theoretical Mathematics from Harvard University. M.S. and PhD in Computer Science from the Courant Institute of Mathematical Sciences, New York University. from the same institution, He developed or was involved with the development of techniques such as 
Perlin noise, real-time interactive character animation, and computer-user interfaces. He is best known for the development of Perlin noise and Simplex noise, both of which are algorithms for realistic-looking Gradient noise. “Received an Academy Award for the development of 
Perlin noise. He had introduced this technique with the goal to produce natural-appearing textures on computer-generated surfaces for motion picture visual effects, while working on the Walt Disney Productions’ 1982 feature film TRON for which he had developed a large part of 
the software.” https://en.wikipedia.org/wiki/Ken_Perlin 

Research interests include graphics, animation, multimedia and science education, VR and AR. 

#person. Timeline: -},9} 

@entry { 

name = {Les Carr} falt-namel = {Leslie Carr} description = {Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and David Millard at the University of Southampton. },§} 

@entry { 

name = {Lev Vygotsky} {description = {was a Russian cognitive scientist, psychologist, constructivist and critical realist whose work focused on the internal mental structure of an individual. Methodologically, he focused on relationships, processes and levels of analysis. He is 
best known for sociocultural theory, a developmental school of thought focused on the relationship between thought and language as independent and dynamic processes in ontogenesis, phylogenesis, and within a cultural context. This dialogue speculates on Vygotsky’s position 
regarding language and virtual reality based on his book Thought and Language. 

Timeline: 1896 — 1934. Born: Russia.},} 

@entry { 

name = {Liquid Information} alt-namel = {liquid} {description = {Frode Hegland’s philosophy of interactive computing. 

Developed while a student in New York, the fundamentals are removing barriers to rich interaction. Later influenced and expanded by the work of his mentor Doug Engelbart and fleshed out in discussions with Sarah Walton. 

https://www.liquidinformation.org} 4} 

@entry { 

name = {LiSA} {description = {Software by Frode Hegland, featuring the voice of Janine Earl. 

Discontinued: http://www.liquid info/lisa/}.§)} 

@entry { 

name = {Livia Polanyi} {description = {is also a contributor to the first volume of The Future of Text. 

Lives: NY, USA.}.} 

@entry { 

name = {Lorenzo Bemaschina} {description = {is a software engineer. 

Founder of Gems, a PKM SaaS to visually manage notes with the help of Al. Interested in tools for thought, human-computer interaction, and everything related to mind- expanding technology. 

Master’s degree in Machine Learning and Artificial Intelligence from Politecnico di Milano. Full-stack developer (MEAN/MERN stack + iOS). 

#person. Timeline: 3 July 1995-, From Lake Como, Italy.}.4)} 

@entry { 

name = {Machine learning} {description = {Machine learning (ML) is a field of inquiry devoted to understanding and building methods that ‘learn’, that is, methods that leverage data to improve performance on some set of tasks. 

A type of Artificial Intelligence (AI). 

https://en.wikipedia.org/wiki/Machine_learning 

#Technology. Timeline: 20th century-.}.4]} 

@entry { 

name = {manuscript} {description = {the authoring format, such as Microsoft Word, which is then either shared as-is, and stays editable, or is exported to be published in a publish format, such as PDF. 

#term},{} 

@entry { 

name = {Map} description = {In this context, a view in the Author software. 

Here the user can place text anywhere they want. If there are defined concepts on the map, the user can click on them and lines will emanate to any text on the map which is in that text’s definition, 

#term},4} 

@entry { 

name = {Mark Anderson} description = {Independent researcher in Hypertext and Knowledge systems. Associated with the Web & Internet Science (WAIS) Lab at Southampton University. 

Mark was heavily involved in the technical specification and ACM launch of Visual-Meta. 

Mark has a background in organisational structure, knowledge, process change and data interchange. 

Co-Editor of the Journal.}.4)} 

@entry { 

name = {Mark Anderson} {description = {is a cocurator of The Future of Text Symposium, 

Independent researcher and consultant in hypertext, knowledge systems, and the retention/support of organisational knowledge. A background in organisational structure, knowledge, process change and data interchange consulting across public, private and NFP sectors. 

A long-term contributor to open projects and online communities since the mid-90s. Also a contributor to various data formats: PDF metadata, IPTC v4 and most recently the technical specification of Visual-Meta including facilitating ACM’s implementation of Visual-Meta. Part of 
hypertext study has included the recovery for use of early hypertext systems otherwise lost to current researchers and creating ebooks for some of Ted Nelson’s key (out of print) books. 

Publications & datasets: https://scholar.google.co.uk/citations?hl=en&user=jn8crEAAAAAT 

Other public resources: https://www.acrobatfag.com 

PhD in Web Science from the University of Southampton: advisors Les Carr and Dave Millard A Visiting Fellow at the University of Southampton, associated with its Web & Internet Science (WAIS) Lab. 

#person. Timeline: 1959 -. Lives in: Portsmouth, UK.},} 

@entry { 

name = {Marvin Minsky}. description = {was an American cognitive and computer scientist concerned largely with research of artificial intelligence (Al), co-founder of the Massachusetts Institute of Technology's AI laboratory, and author of several texts concerning Al and 
philosophy. 

https://en.wikipedia.org/wiki/Marvin_Minsky 

#person Timeline: 9 August 1927 ~ 24 January 2016},4} 

@entry { 

name = {Mesopotamia} {description = {a historical region within the Tigris-Euphrates river system in the northern part of the Fertile Crescent where early writing developed (in parallel to, or influenced by/influencing Egyptian writing), including the use of colophons.},§} 
@entry { 

name = {Meta} {description = {is a large social media company which is focusing on what they term the ‘metaverse’. 

Formerly called Facebook. 

https://about.facebook.com 

#Institution. Timeline: 4 February 2004-.} 4} 

@entry { 

name = {metadata} {description = {information about other information, in the case of documents, this can include structural information (headings for example), biblio.} 4} 

@entry { 

name = {Metaverse} {description = {From Wikipedia: “A metaverse is a network of 3D virtual worlds focused on social connection” 

“The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow Crash” 

https://en.wikipedia.org/wiki/Metaverse 

At the time of writing, 2022, the term has been popularised by Meta, the company previously called Facebook. Most often used in conjunction with AR/VR augmented environments, not such much the 3D worlds where the user access the world through a flat screen. }]} 
@entry { 

name = {Mez Breeze} description = {is an Expert and Innovator of contemporary digital culture. 

Since the early 1990’s, Mez Breeze has published over 300 seminal works including award-winning electronic based writing, Virtual Reality literature, Artificial Intelligence artworks/projects, books, games, and other genre-defying output all while teaching and supporting digital 
art and electronic literature. In April 2022, Mez’s Artificial Intelligence Artwork ‘Post Glee[son] - Outside [R]’ (created through digitally stitching GauGAN and VQGAN+CLIP output) made the finals of the 2022 Goulburn Art Award. In July 2019, Mez won the 2019 Marjorie C. 
Luesebrink Career Achievement Award which: “...honors a visionary artist and/or scholar who has brought excellence to the field of electronic literature and has inspired others to help create and build the field.” Mez's projects are taught worldwide with her works residing in 
Collections as diverse as The World Bank, Cornell's Rose Goldsen Archive and the National Library of Australia. She currently serves as an Advisor to the Mixed Augmented Reality Art Research Organisation, an Editorial Board Member of the Digital Journal Thresholds, and is a 
Senior Research Affiliate of the Humanities and Critical Code Studies Lab. 

Her latest book ‘[Por]TrAlts: AI Characters + Their Microstories’ (written in her trademark Mezangelle style while incorporating AI output) was a Top Seller on the publishing platform Itchio in September 2022. 

https://www.mezbreezedesign.com, https://mezbreeze..itch.io/portraits-volume-one 

#person. Timeline: 1970-. Company: Mez Breeze Design. Lives in: Australia. Key Interests: Al. VR. XR.}.4)} 

@entry { 

name = {Michael Roberts} {description = {is a Scientist Mike Roberts currently resides in the Santa Cruz Mountains, California, USA. 

He received his doctorate in the early 90’s, working on the intersection between parallel computing and no-code visual programming, specifically defining an isomorphism between Tony Hoare’s Communicating Sequential Processes (CSP) calculus and graph theory. 

After moving the US, he began work on tools for interactive, creative, computer-based media, specifically targeting computer graphics, multimedia, augmented and virtual realities. His work has influenced the technical and business direction of many US-based fortune 500 
companies including Xerox, Autodesk, Progress Software, IPS/Motorola as well as prominent Japanese companies like DNP. 

Prior to founding ConstuctiveLabs, his current metaverse company, he recently spent 9.5 years at renowned research lab Xerox PARC, where he worked on projects including conversational agents, contextual and artificial intelligence, and AR / VR. His interests in tool-making for 
creatives are informed by studies in traditional tool ecosystems and slow food production, 

Founder of Constructivelabs. 

#person. }.4)} 

@entry { 

name = {Murals} f[alt-namel = {mural} {description = {This is a special entry to experiment with layouts. },§} 

@entry { 

name = {NIC} {description = {Doug Engelbart concept: Networked Improvement Community “Consider an "Improvement Community" (IC) as collectively engaged in improving an agreed-upon set either of individual capabilities, or of collective group capabilities-e.g. a 
professional society. Let's introduce a new category, a "Networked Improvement Community" (NIC): an IC that is consciously and effectively employing best-possible DKR (Dynamic Knowledge Repository) development and usage.” 

(augmenting society's collective 1Q).}.4]} 

@entry { 

name = {NLS} {description = {From Wikipedia: “NLS, or the "oN-Line System", was a revolutionary computer collaboration system developed in the 1960s. Designed by Douglas Engelbart and implemented by researchers at the Augmentation Research Center (ARC) at the 
Stanford Research Institute (SRI), the NLS system was the first to employ the practical use of hypertext links, the mouse, raster-scan video monitors, information organized by relevance, screen windowing, presentation programs, and other modern computing concepts. It was 
funded by ARPA (the predecessor to Defense Advanced Research Projects Agency), NASA, and the US Air Force.” https://en.wikipedia.org/wiki/NLS_(computer_system) 

SRI sold NLS to Tymshare in 1977 and renamed it Augment.} 4} 

@entry { 

name = {Norway} {description = {https://en..wikipedia.org/wiki/Norway 


#Location},§} 


@entry 

name = {NYU} {description = {(New York University) is a private research university in New York City. Chartered in 1831 by the New York State Legislature, NYU was founded by a group of New Yorkers led by then-Secretary of the Treasury Albert Gallatin. 
https://en.wikipedia.org/wiki/New_York_University},{]} 

@entry | 


name = {Omar Rizwan} {description = {Contributor to the Journal. 

Omar has been interested in new computer interfaces and new ways of programming (aren't these the same thing?). He has worked at Dynamicland, at Stripe, and at Khan Academy. 
i am determined to move beyond this way of interacting with systems 

Among other things, I'm the creator of +, ,Sereenotate, a tool for macOS and Windows which captures the text and origin (URL, window title, ...) whenever you take a screenshot. 


https://screenotate.com/ 


hitps://twitter.com/rsnous 
https://omar.website 
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@ontry 

name = {Open Al} {description = {is an artificial intelligence (AI) research laboratory consisting of the for-profit corporation OpenAl LP and its parent company, the non-profit OpenAl Inc. 

Products: DALL-E, GPT-3, GPT-2. OpenAl Gym 

https://openai.com/ 

https://en.wikipedia.org/wiki/OpenAl} 4} 

@ontry 

name = {OpenAl} {description = {is an artificial intelligence (Al) research laboratory consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAl Ine. 

Products: GPT-1. GPT-2. GPT-3 

https://openai.com 

https://en.wikipedia.org/wiki/OpenAl} 4} 

@entry 

name = {OpenAl} {description = {is an artificial intelligence (Al) research laboratory. 

“...consisting of the for-profit corporation OpenAlI LP and its parent company, the non-profit OpenAl Inc. 

The company, considered a competitor to DeepMind, conducts research in the field of AI with the stated goal of promoting and developing friendly AI in a way that benefits humanity as a whole. The organization was founded in San Francisco in late 2015 by Elon Musk, Sam 
Altman, and others, who collectively pledged US$1 billion. Musk resigned from the board in February 2018 but remained a donor. In 2019, OpenAl LP received a USS! billion investment from Microsoft.” 

https://en.wikipedia.org/wiki/OpenAI 

https://openai.com 

Products include: GPT-3 and DALL-E. 

Founded by Elon Musk, Sam Altmanllya Sutskever, Greg Brockman, Wojciech Zaremba and John Schulman. 

Timeline: 11 December 2015-.}.4)} 

@entry 

name = {paper} description = {is a general term for a student or academic document in general. 

Primarily in PDF when published or handed in. 

In manuscipt/editable/personal form it is generall in the Microsoft Word format.},)} 

@entry| 

name = {PDF} description = {*Portable Digital Format’. Developed by Adobe. Now free with no license restrictions. It is a print to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for users to interact with the document in useful 
ways, while staying compatibel with the basic PDF format. 

#Infrastructure. Timeline: Introduced 15 June 1993-.},4} 

@ontry 

name = {Peter Wasilko} description = {is an Attorney, Programmer, and Indepedent Scholar. 

He holds a J.D., LL.M., and Certificate in Law, Technology, and Management from Syracuse University’s College of Law. He is admitted to practice law in New York State and is a member of the New York State Bar Association and its Intellectual Property Section. He also 
maintains memberships in the ACM, IEEE Computer Society, and Association for the Advancement of Aritificial Intelligence. His primary technical interests are Hypertext, Intelligence Augmentation, Human-Computer Interaction, Computer Supported Cooperative Work, 
Symbolic Al, Law and Al, and Programming Language Design. His intertests beyond computing focus on Foresight and Futures Studies with an emphasis on University Futures and forays into a wide range of related disciplines ranging from Innovation Studies, Design Futures and 
Speculative Design; Medieval Universities and Trade Guilds; World’s Fairs and Theme Parks; TechnoCities and Prototype Communities of Tomorrow; Architecture; and Urban Planning. He founded Founders’ Quadrangle — an unincorporated association of academies explorng the 
design space for Universities and Quasi Academic Enterprises of the Future. He can be found with email, LinkedIn, ORCID iD, and Twitter. 

#person. Timeline: -, Lives in: New York, New York, USA.},f]} 

@entry 

name = {Pol Baladas} {description = {is the CEO and Artisan at Fermat, a computational medium where people can build their own tools and use tools built by others (we're basically bringing back HyperCard). 

Fermat is made inside Batou.xyz, an industrial research lab that rethinks how people interact with tools, computers and ideas. Interested in tools, media, thought and a better future, Computing philosopher and charlatan. 

Interested in Tools, thought, interfaces and a better future. 

https://twitter.com/polbaladas 

person. Timeline: -},} 

@entry | 

name = {Quest 2}. {description = {The headset we currently use the most in the Future Text Lab. It sold more than Xbox in 2021 and next year we will see the Apple HMD which will enlarge the field of VR/AR even further. 

#Hardware | Timeline: 13 October 2020- | Company: Meta 

iil 

@ontry 

name = {Reader} (description = {a PDF viewer from the Augmented Text company. 

https://www.augmentedtext.info/reader} 4) 

@entry 

name = {Reader Software} description = {is a minimalist PDF viewer for macOS. 

Can read any PDF and can provide added interactions if the PDF has Visual-Meta attached, This can either be produced by Author or any other word processor with Visual-Meta capability, or downloaded from an online repository which features Visual-Meta, such as the ACM 
digital library. Produced by The Augmented Text Company LTD, with programming by Jacob Hazelgrove. 

https://www.augmentedtext.info 

#Software. Timeline: 12 July 2019-}.4)} 

@entry 

name = {references} {description = {is a list of all the citations a document uses, in an Appendix. In-Body citation, point to these References. This language is not fixed, it is sometimes used interchangeably with Bibliography but in my context a Bibliography is a list of work not 
expressly cited but which are relevant. 

in this context ‘Reference’ with uppercase *R’ refers to the appendix in an academic document which lists cites sources. In contrast, the citation in the body of the document is referred to as in-body citation, 

‘A form of metadata. 

Herm}. 

@entry 

name = {Relative Dimension} {description = {A dimension capturing the ordering of data objec 
@ontry 

name = {Result Set}, {description = {The set of data object satisfying a database search query.}.§} 

@entry 

name = {Richard Snyder} description = {Electronic Literature Lab, Washington State University Vancouver. 

Contributed to this book with Dene Grigar.}.4} 

@entry 

name = {Russia} {description = {or the Russian Federation, is a transcontinental country spanning Eastern Europe and Northern Asia. It is the largest country in the world.},} 

@ontry 

name = {Sam Brooker} description = {I am an academic specialising in digital communication, His research explores the relationship between digital technologies & theories of literature and culture. 

Associate Professor of Digital Communications at Richmond American University London, UK. 

His research explores the relationship between digital technologies & theories of literature and culture. He is a regular contributor to ACM Hypertext and Social Media, frequently serving on the programme committee. Additional contributions have featured at the ICIDS and 
SHARP conferences. I am a member of the PRCA International University Advisory Group and academic reviewer for numerous journals. 

https://www.richmond.ac.uk/school-of-communications-arts-social-sciences/dr-sam-brooker/ 

person. Timeline: -, Research interests: Digital cultures. Electronic literature. Book history. Digital humanities, Theories of authorship. Transmedia. Lives in: London, UK.}.4} 

@entry 

name = {Scott Rettberg} description = {is professor of digital culture in the department of linguistic, literary, and aesthetic studies at the University of Bergen, Norway. 

Rettberg was the project leader of ELMCIP (Electronic Literature as a Model of Creativity and Innovation in Practice), a €1.000.000, six nation, HERA-fuunded collaborative research project, from 2010-2013. 

Leader of the Bergen Electronic Literature Research Group. Director of the ELMCIP Electronic Literature Knowledge Base. Rettberg is the author or coauthor of novel-length works of electronic literature, combinatory poetry, and films including The Unknown, Kind of Blue, 
Implementation, Frequency, The Catastrophe Trilogy, Three Rails Live, Toxi*City, Hearts and Minds: The Interrogations Project and others. 

His creative work has been exhibited online and at art venues including the Venice Biennale, Santa Monica Museum in Barcelona, the Inova Gallery, Rom 8, the Chemical Heritage Foundation Museum, Palazzo dell Arti Napoli, Beall Center, the Slought Foundation, The Krannert 
‘Art Museum, and elsewhere, Cofounder and served as the first executive director of the nonprofit Electronic Literature Organization, where he directed major projects funded by the Ford Foundation and the Rockefeller Foundation. Rettberg served on the ELO board of directors 
from 2001-2015, Rettberg’s Electronic Literature (Polity, 2018) is the first comprehensive study of the history and genres of electronic literature. Rettberg was recently awarded a SAMKUL grant from Research Council of Norway to lead a four year research project “Extending 
Digital Narrative.” 

hitps://twitter.com/scottrettberg 

person. Timeline: 1970. Born: Norway. Lives in: Bergen, Norway.} 4) 

@entry 

name = {Search} {description = {Search in this context refers to instant search enabled by Liquid. }.4} 

@ontry 

name = {Softspace} {description = {*Softspace is inventing a new kind of tool for thought for thinkers and makers using virtual and augmented reality.” 

A tool for thought. 

https://soft.space 

#Software | Timeline: -. Keywords: VR. AR.}.4} 

@ontry 

name = {Spain} {description = {https://en.wikipedia.org/wiki/Spain 

#Hlocation. Timeline: -. }.4} 

@ontry 

name = {Spatial Computing} (description = {“was defined in 2003 by Simon Greenwold, as “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces”. With the advent of consumer VR, AR and mixed reality, companies 
use ‘spatial computing’ in reference to the practice of using physical actions (head and body movements, gestures, speech) as inputs for interactive digital media systems, with perceived 3D physical space as the canvas for video, audio, and haptic outputs. It is also tied to the 
concept of ‘digital twins’. 

https://en.wikipedia.org/wiki/Spatial_computing 

il 

@entry { 

name = {SSTV} {description = {(Slow-Scan Television) is a picture transmission method, used mainly by amateur radio operators, to transmit and receive static pictures via radio in monochrome or color. 

https://en.wikipedia.org/wiki/Slow-scan_television}.q)} 


@entry 


based on the pairwise application of a comparison function. 4} 


name = {stable diffusion} {description = {is a deep learning, text-to-image model released by startup StabilityAI in 2022. 
Keywords: collective intelligence. augmented technology. Al. 
#technology. 


hitps://en.wikipedia.org/wiki/Stable_Diffusion https://stability.ai} 4} 
@entry { 


name = {Stephen Fry} description = {is an English actor, broadcaster, comedian, director and writer. 

Contributor also to volume | of The Future of Text. 

Timeline: 24 August 1957-. Born: UK},} 

@entry { 

name = {Stigmergy} {description = {(/'stigmardgi/ STIG-mor-jee) is a mechanism of indirect coordination, through the environment, between agents or actions. 

The principle is that the trace left in the environment by an individual action stimulates the performance of a succeeding action by the same or different agent. Agents that respond to traces in the environment receive positive fitness benefits, reinforcing the likelihood of these 
behaviors becoming fixed within a population over time. 

https://en.wikipedia.org/wiki/Stigmergy 

#eoncept 

Mh 

@entry { 

name = {Ted Nelson} alt-namel = {Theodor Holm Nelson} {description = {is an American pioneer of information technology, philosopher, and sociologist. 

He coined the term ‘hypertext’, 

Presented at The Future of Text Symposium. 

#person. Timeline: 17 June 1937-.},4} 

@entry { 

name = {Text} description = {is the basic “stuff” of this work.},4)} 

@entry { 

name = {The Future of Text} {description = {Annual Symposium and Book Series (first volume published 2020) as well as community for fostering dialogue around the future of text which I started over a decade ago and which is often co-hosted or presented by Vint Cerf. 
https://futuretextpublishing.com} §}} 

@entry { 

name = {The Future of Text Symposium} {description = {is a series of symposia, books and Journal under the name ‘The Future of Text’ produced by the same people who run The Augmented Text Company. 

Co-presented by Frode Hegland, Vint Cerf and Ismail Serageldin. Curated by Dene Grigar, Claus Atzenbeck and Mark Anderson. 

This is considered as level C of Doug Engelbart’s Three Levels of Activity. 

https://futuretextpublishing.com 

#Event | Timeline: 2011-},§} 

@entry { 

name = {Tom Standage} {description = {Deputy Editor of The Economist. 

Contributor to The Future of Text volume 1. 

Born: UK.}.j} 

@entry { 

name = {tool for thought} description = {is both a book and a category of tools, primarily software tools, to augment how people think. 

The book: “Tools for Thought: The History and Future of Mind-Expanding Technology is a work of "retrospective futurism"” in which Smart Mobs author Howard Rheingold looked at the history of computing and then attempted to predict what the networked world might look like 
in the mid-1990s. The book covers the groundbreaking work of thinkers like Alan Turing, John von Neumann, and J.C.R. Licklider, as well as Xerox PARC, Apple Computer, and Microsoft (when Microsoft was "aiming for the hundred-million-dollar category"). Rheingold wrote 
that the impetus behind Tools for Thought was to understand where "mind-amplifying technology” was going by understanding where it came from.” https://en.wikipedia.org/wiki/Tools_for_Thought}.} 

@entry { 

name = {UK}. description = {https://en.wikipedia.org/wiki/United_Kingdom 

#Location},§} 

@entry { 

name = {Ukraine} {description = {https://en.wikipedia.org/wiki/Ukraine 

#Location}.§} 

@entry { 

name = {University of Southampton} {description = {University in the UK. 

#Institution. Timeline: 1862-. Southampton, UK} 4} 

@entry { 

name = {USA} {description = {https://en.wikipedia.org/wiki/United_States 

#Location} §} 

@entry { 

name = { ViewSpec} [description = {Engelbart Concept: View Specifications for a user to see their work in different views. 

#dougconcept #term} 4} 

@entry { 

name = {Vint Cerf} falt-namel = {Vinton Gray Cerf} fjalt-name2 = {Vinton G. Cerf} {description = {is cohost of the Future of Text Symposium. 

Coinventor of the Internet, VP and Chief Internet Evangelist for Google. Chairman of the Marconi Society. Former executive at MCI, the Corporation for National Research Initiatives, the Internet Corporation for Assigned Names and Numbers (ICANN), the American Registry for 
Internet Numbers (ARIN), the Association for Computing Machinery (ACM) and member of the Faculty of Stanford University. Fellow of IEEE, ACM, BCS, AAAS, American Academy of Arts and Sciences, and American Philosophical Society. Member of the US National 
Academies of Engineering and Science and foreign member of the Royal Society and the Royal Swedish Engineering Society. 

#person. Timeline: 23 June 1943-.},4} 

@entry { 

name = { Visual-Meta} {description = {An open and robust way to augment flat PDF documents to make them more interactive. 

Frode Hegland’s PhD thesis. Realised in the Author software and Reader software by The Augmented Text Company. Intended to support Doug Engelbart’s notion of hyperdocuments and xfiles. 

Inspired by the BibTeX format and the colophon in books. 

http:/visual-meta.info 

#Infrastrueture} } 

@entry { 

name = { Visual-Meta} {description = {A open and robust way to augment flat PDF documents to make them more interactive, by The Augmented Text Company people. 

http:/visual-meta.info} 4} 

@entry { 

name = {VR} {description = {VR, ‘Virtual Reality’ is a technology whereby the user is visually immersed in a computer generated world. When this world allows the physical world to also appear as background, it is refered to as AR ‘Augmented Reality’. 
#Technology | Timeline: 20th century-},4} 

@entry { 

name = {VR} (description = {VR, ‘Virtual Reality’ is a technology whereby the user is visually immersed in a computer generated world. When this world allows the physical world to also appear as background, it is called AR ‘Augmented Reality’. 
#Technology | Timeline: 20th century-},4} 

@entry { 

name = {Yiliu Shen-Burke} {description = {Yiliu Shen-Burke is a designer and coder. 

He is building Softspace, a VR / AR knowledge-mapping tool for creative people (https://www.soft.space). Previously, he was a research resident at Studio Olafur Eliasson in Berlin, He has a B.A. in econometrics from Columbia University, and dropped out of the Harvard 
Graduate School of Design. He creates out of a love for making beautiful things that elevate those who use them. 

https://twitter.comvyiliu_shenburke https://www.yiliu.sh 

He is featured in issue 1.4 of our Journal: https://futuretextpublishing.com/journal/ 

#Person. Timeline: - Keywords: tool for thought. Spatial computing. } 4} 


@{glossary-end} 
@{document-headings-start} 


@heading{ 

name = {Foreword} level = {level} fauthor = {Frode Alexander Hegland} 4} 
@heading{ 

name = {by Vint Cerf} level = {level2} jauthor = {Frode Alexander Hegland} 4} 


@heading{ 

name = {Welcome} ,flevel = {level} fauthor = {Frode Alexander Hegland} } 
@heading{ 

name = {by Frode Hegland} level = {level2}, author = {Frode Alexander Hegland} 4} 
@heading{ 


name = {Our work in VR} flevel = {level2},jauthor = {Frode Alexander Hegland} } 
@heading{ 
name = {This Book as Augmented PDF} level = {level2} ffauthor = {Frode Alexander Hegland) 4} 


@heading{ 

name = {Editor’s Introduction} level = {level2} author = {Frode Alexander Hegland} 4} 
@heading{ 

name = {Andreea lon Cojocaru} [level = {level } author = {Andreea Ion Cojocaru}.]} 
@heading{ 


name = {Borges and Vygotsky Join Forces for BOVYG, Latest Virtual Reality Start-up} {level = {level2} {author = {Andreea Ion Cojocaru} 4j} 
@heading{ 
name = {Abstract} fflevel = {level3} {author = {Andreea Ion Cojocaru} i} 


@heading{ 

name = {Body} level = {level3} author = {Andreea lon Cojocaru}.j} 
@heading{ 

name = {Author's Notes},fflevel = {level3} author = {Andreea Ion Cojocaru} 4} 
@heading{ 


name = {Journal Guest Presentation ‘An Architect Reads Cognitive Neuroscience and Decides to Start an Immersive Tech Company’ : 13 May 2022},fllevel = {level3} author = {Andreea Ion Cojocaru} 4} 
@heading{ 
name = {Q&A}. fllevel = {level3} author = {Andreea Ion Cojocaru}.} 


@heading{ 

name = {Andy Campbell} level = {level} fjauthor = {Andy Campbell} } 

@heading{ 

name = {Dreaming Methods - Creating Immersive Literary Experiences} level = {level2} author = {Andy Campbell} ]} 
@heading{ 


name = {Presentation (pre-recorded for the Symposium)} fflevel = {level3} {fauthor = {Andy Campbell} §} 


@heading{ 

name = {Annie Murphy Paul} level = {level } author = {Annie Murphy Paul} ]} 

@heading{ 

name = {Operationalizing the Extended Mind) level = {level2} author = {Annie Murphy Paul},§} 
@heading{ 

name = {Apurva Chitnis} fflevel = {levell } author = {Apurva Chitnis} 4)} 

@heading{ 

name = {Journal : Public Zettlekastens} {level = {level2} author = {Apurva Chitnis} 4} 
@heading{ 

name = {Limitations today} {level = {level3} author = {Apurva Chitnis}.]} 


@heading{ 

name = {Public Zettlekiisten} fJlevel = {level3} §Jauthor = {Apurva Chitnis}.)} 
@heading{ 

name = {Implementation} level = {level3} fauthor = {Apurva Chitnis} ]} 
@heading{ 

name = {Challenges} {level = {level3} author = {Apurva Chitnis}.]} 
@heading{ 

name = {Barbara Tversky} level = {levell } author = {Barbara Tversky}4)} 
@heading{ 


name = {Journal Guest Presentation : Mind in Motion} level = {level2} author = {Barbara Tversky}.4]} 
@heading{ 

name = {Q&A}. fllevel = {level3} author = {Barbara Tversky}.4]} 

@heading{ 

name = {Bjorn Borud} level = {level } author = {Bjorn Borud} } 

@heading{ 

name = {Time, speed and distance} {level = {level2} fjauthor = {Bjorn Borud} )} 


@heading{ 

name = {Computers and light speed} level = {level3} author = {Bjorn Borud),} 
@heading{ 

name = {Signal strength and distance} fflevel = {level3} author = {Bjorn Borud} 4} 
@heading{ 


name = {The Drake equation} level = {level3} fauthor = {Bjorn Borud).} 
@heading{ 
name = {Our civilization} fflevel = {level3} author = {Bjom Borud} 4} 


@heading{ 

name = {Bob Horn} filevel = {levell } {author = {Bob Horn}.4]} 

@heading{ 

name = {Information Murals for Virtual Reality} filevel = {level2} author = {Bob Horn} 4} 
@heading{ 


name = {Introduetion: my recent work} {level = {level3} ffauthor = {Bob Horn},} 
@heading{ 
name = {My role as synthesizer} level = {level3} author = {Bob Horn} } 


@heading{ 

name = {Examples of Information Murals} fflevel = {level3} Jauthor = {Bob Horn} qj} 
@heading{ 

name = {Overwhelmed by complexity?} [level = {level3} author = {Bob Horn} } 
@heading{ 

name = {Why am I here at this Symposium?} level = {level3} author = {Bob Horn},§} 
@heading{ 

name = {Text as idea chunks with subheads} {level = {level3} fjauthor = {Bob Horn},} 
@heading{ 

name = {Benefits of small idea chunks with subheads} fjlevel = {level3} author = {Bob Horn}. } 
@heading{ 

name = {Transition to other offerings} fflevel = {level3} author = {Bob Horn} .j} 
@heading{ 


name = {Assumption: improve human thinking} level = {level3} author = {Bob Horn},§} 
@heading{ 

name = {What can we do to move toward Einstein’s goal?},f[level = {level3} author = {Bob Horn} j} 
@heading{ 

name = {Problem: Show and link context} fllevel = {level3} author = {Bob Horn} 4} 

@heading{ 

name = {Show and link context. ..in Multiple Dimensions} fflevel = {level3} author = {Bob Horn} j} 
@heading{ 

name = {Problem: Show process visually} [level = {level3} fauthor = {Bob Horn},{} 

@heading{ 

name = {Problem: build solid and supportive “scaffoldings for thinking”} fflevel = {level3} author = {Bob Horn} j} 
@heading{ 

name = {Offer of help},fflevel = {level3} author = {Bob Horn} §)} 


@heading{ 

name = {Bibliography/Further Reading} level = {level3} author = {Bob Hor},§} 
@heading{ 

name = {Bob Stein} fflevel = {levell } author = {Bob Stein}.]} 

@heading{ 


name = {Journal Guest Presentation : 4 July 2022} level = {level2} author = {Bob Stein} §j} 
@heading{ 
name = {Screenshots} fflevel = {level3} author = {Bob Stein} ]} 


@heading{ 

name = {Brett Jackson} level = {level} author = {Brett Jackson} i} 

@heading{ 

name = {The evolution of mind maps for interactive VR experiences} fflevel = {level2} author = {Brett Jackson) ,{} 
@heading{ 

name = {Caitlin Fisher} [level = {level } author = {Caitlin Fisher} 9} 

@heading{ 

name = {Daveed Benjamin} level = {levell } author = {Daveed Benjamin} } 
@heading{ 

name = {Thoughts about Metadata} {level = {level2} fauthor = {Daveed Benjamin} 4} 
@heading{ 


name = {Cynthia Haynes & Jan Rune Holmevik} level = {levell } {author = {Cynthia Haynes and Jan Rune Holmevik}.} 
@heading{ 

name = {Teleprompting Elekcriture} [level = {level2} jauthor = {Cynthia Haynes and Jan Rune Holmevik} 4} 
@heading{ 

name = {Works Cited} level = {level3 } author = {Cynthia Haynes and Jan Rune Holmevik},} 

@heading{ 

name = {Deena Larsen},fjlevel = {level } fjauthor = {Deena Larsen} qj} 

@heading{ 

name = {Access within VR: Opening the Magic Doors to All},fjlevel = {level2} author = {Deena Larsen} qj} 
@heading{ 

name = {Dene Grigar & Richard Snyder} level = {levell } author = {Dene Grigar and Richard Snyder} §)} 
@heading{ 

name = {Metadata for Access: VR and Beyond} level = {level2) author = {Dene Grigar and Richard Snyder} .]} 
@heading{ 

name = {Eduardo Kac} fjlevel = {level} author = {Eduardo Kac} }} 

@heading{ 

name = {Space Art: My Trajectory} level = {level2} Jauthor = {Eduardo Kac},} 

@heading{ 

name = {Introduction} fflevel = {level3} fauthor = {Eduardo Kac} 4} 

@heading{ 

name = {Agora: a holopoem to be sent to Andromeda},fllevel = {level3} fjauthor = {Eduardo Kac}.j} 
@heading{ 

name = {Spacescapes} level = {level3} author = {Eduardo Kac} 4} 

@heading{ 

name = {Monogram} filevel = {level3} author = {Eduardo Kac},{} 

@heading{ 

name = {The Lepus Constellation Suite} level = {level3} author = {Eduardo Kac}.j} 


@heading{ 
name = {Lagoogleglyphs} fflevel = {level3} {author = {Eduardo Kac} ]} 
@heading{ 
name = {Inner Telescope} fflevel = {level3} {author = {Eduardo Kac} 4} 
@heading{ 


name = {Adsum, an artwork for the Moon} level = {level3} author = {Eduardo Kac} 4} 
@heading{ 
name = {Conclusion} fflevel = {level3} author = {Eduardo Kac}.]} 


@heading{ 
name = {Fabien Benetou},fjlevel = {level} {author = {Fabien Benetou}.4]} 


@heading{ 
name = {Why PDF is the wrong format to bring text to XR and why the Web with proper provenance and responsive design from stylesheets is all we need} [level = {level2} author = {Fabien Benetou} 4)} 
@heading{ 

name = {Fabien Benetou},flevel = {level } author = {Fabien Benetou} Jj} 

@heading{ 


name = {The Case Against Books} level = {level2}, author = {Fabien Benetou}.]} 
@heading{ 
name = {Fabien Benetou},fjlevel = {level } ffauthor = {Fabien Benetou}.4)} 


@heading{ 

name = {Interfaces all the way down} [level = {level2} author = {Fabien Benetou} 4} 
@heading{ 

name = {Fabien Benetou}, level = {level } ffauthor = {Fabien Benetou} 4j} 
@heading{ 


name = {Stigmergy Across Media} ‘level = {level2} author = {Fabien Benetou},{} 
@heading{ 
name = {Fabien Benetou},fllevel = {level } {author = {Fabien Benetou}.4)} 


@heading{ 

name = {Journal : Utopiah/visual-meta-append-remote,js} level = {level2} author = {Fabien Benetou} .j} 
@heading{ 

name = {Frode Hegland) level = {level } author = {Frode Hegland} 4} 

@heading{ 


name = {The state of my text art + the journey to VR} level = {level2} author = {Frode Hegland} 4} 
@heading{ 
name = {State of the my art}, llevel = {level2} author = {Frode Hegland} §j} 


@heading{ 

name = {Editing} {level = {level2} fjauthor = {Frode Hegland}.)} 
@heading{ 

name = {Research}, fflevel = {level2} author = {Frode Hegland} §j} 
@heading{ 


name = {Making it happen} level = {level2} author = {Frode Hegland} J} 
@heading{ 
name = {Frode Hegland} flevel = {level } author = {Frode Hegland} 4} 


@heading{ 

name = {The case for books} level = {level2} author = {Frode Hegland) 4} 
@heading{ 

name = {Robustness} fflevel = {level3} §jauthor = {Frode Hegland} 4} 
@heading{ 


name = {Book Bindings} level = {level2} jauthor = {Frode Hegland),} 
@heading{ 
name = {Digital Bindings} ffJevel = {level2} {author = {Frode Hegland} 4)} 


@heading{ 

name = {Future Books} {level = {level3} {author = {Frode Hegland} §} 
@heading{ 

name = {Frode Hegland} level = {level } author = {Frode Hegland} 4} 
@heading{ 

name = {*Just’ more displays?},flevel = {level2} author = {Frode Hegland} }} 
@heading{ 

name = {Frode Hegland} level = {level } author = {Frode Hegland} 4} 
@heading{ 

name = {Page to Page Navigation} level = {level2}, author = {Frode Hegland} j} 
@heading{ 

name = {Frode Hegland} ,flevel = {level } author = {Frode Hegland} 4} 
@heading{ 


name = {Journal : Academic & Scientific Documents in the Metaverse} [level = {level2} jauthor = {Frode Hegland} } 
@heading{ 
name = {Jack Kausch} fflevel = {level } author = {Jack Kausch},§} 


@heading{ 

name = {Why We Need a Semantic Writing System} level = {level2} {author = {Jack Kausch}.j} 
@heading{ 

name = {Jad Esber} fflevel = {levell} fauthor = {Jad Esber} §)} 

@heading{ 


name = {Monthly Guest Presentation : 21 February 2022} {level = {level2} author = {Jad Esber} j} 
@heading{ 
name = {Dialogue},fflevel = {level3} author = {Jad Esber},} 


@heading{ 

name = {Gavin Menichini} {level = {levell } author = {Jad Esber} } 

@heading{ 

name = {Journal Guest Product Presentation : 25 February 2022} level = {level2} ffauthor = {Jad Esber},{} 
@heading{ 


name = {Chat Log} level = {level3} author = {Jad Esber} 4} 
@heading{ 
name = {Harold Thimbleby} level = {levell } {author = {Harold Thimbleby}.} 


@heading{ 
name = {Getting mixed text right is the future of text} {level = {level2} {author = {Harold Thimbleby} 4)} 
@heading{ 

name = {The author’s experience of text} fflevel = {level3} author = {Harold Thimbleby},} 

@heading{ 


name = {Interesting aside....} {level = {level3} {author = {Harold Thimbleby} } 
@heading{ 
name = {Mixed texts in single systems} [level = {level3 } author = {Harold Thimbleby} 4} 


@heading{ 

name = {Future text mixed with AI and ...} {level = {level3} {jauthor = {Harold Thimbleby},{} 
@heading{ 

name = {Conclusions} fflevel = {level3} author = {Harold Thimbleby} 4} 

@heading{ 


name = {Jamie Joyce} fflevel = {level } author = {Jamie Joyce} 4} 
@heading{ 
name = {Guest Presentation : The Society Library} level = {level2} author = {Jamie Joyce},§} 


@heading{ 

name = {Dialogue},fjlevel = {level3} author = {Jamie Joyce} 4} 
@heading{ 

name = {Jaron Lanier} f[level = {level } author = {Jaron Lanier},} 
@heading{ 


name = {Keynote} fjlevel = {level2} fauthor = {Jaron Lanier}.j} 
@heading{ 
name = {Q&A} level = {level3} author = (Jaron Lanier} } 


@heading{ 

name = {Jim Strahorn} level = {level } author = {Jim Strahorn} 4} 

@heading{ 

name = {The Future of ... More Readable Books ... a Reader Point of View) [level = {level2}.author = {Jim Strahorn},]} 
@heading{ 


name = {Kalev Leetaru} level = {levell } fJauthor = {Kalev Leetaru},§} 
@heading{ 
name = {Seeing Through Others’ Eyes: Reimagining How We Experience The News} [level = {level2} author = {Kalev Leetaru} 4} 


@heading{ 

name = {Globalization} level = {level3},jauthor = {Kalev Leetaru},{} 

@heading{ 

name = {From Firehose To Awareness} {level = {level3} author = {Kalev Leetaru},{} 
@heading{ 

name = {Falsehoods} level = {level3} fjauthor = {Kalev Leetaru} 4} 

@heading{ 

name = {Our Ever-Evolving Language} fflevel = {level3} author = {Kalev Leetaru}.)} 
@heading{ 

name = {Preservation} level = {level3} author = {Kalev Leetaru} 4} 

@heading{ 

name = {Interface} level = {level3} {fauthor = {Kalev Leetaru},§} 

@heading{ 


name = {Merging Human & Machine Intelligence} fflevel = {level3} author = {Kalev Leetaru},)} 
@heading{ 
name = {Search} fflevel = {level3} author = {Kalev Leetaru}.4]} 


@heading{ 
name = {Synthesis} {level = {level3} author = {Kalev Leetaru},§} 


@heading{ 

name = {Dimensionality} level = {level3} author = {Kalev Leetaru}.)} 
@heading{ 

name = {Interpretation & Emotion} fflevel = {level3} author = {Kalev Leetaru}.§j} 
@heading{ 

name = {Transformation} fflevel = {level3} ffauthor = {Kalev Leetaru},§} 
@heading{ 

name = {Representation} fflevel = {level3} author = {Kalev Leetaru},]} 
@heading{ 

name = {Ken Perlin} level = {levell } author = {Ken Perlin},} 

@heading{ 

name = {Closing Keynote: Experiential Computing and the Future of Text} level = {level2} author = {Ken Perlin},} 
@heading{ 


name = {Presentation} fflevel = {level3} {author = {Ken Perlin} } 
@heading{ 
name = {Q&A} fllevel = {level3} author = {Ken Perlin},} 


@heading{ 
name = {Livia Polanyi} level = {level} author = {Livia Polanyi}.,4)} 
@heading{ 
name = {Virtual Vision} level = {level2} author = {Livia Polanyi} ,§)} 
@heading{ 


name = {Lorenzo Beraschina} level = {levell } author = {Lorenzo Bernaschina} 4}} 

@heading{ 

name = {Gems} level = {level2} jauthor = {Lorenzo Bernaschina} j} 

@heading{ 

name = {Mark Anderson}. [level = {level } author = {Mark Anderson},§} 

@heading{ 

name = {Image Maps and VR: not as simple as supposed}, level = {level2} author = {Mark Anderson} 4} 
@heading{ 

name = {Abstract} fflevel = {level3} {author = {Mark Anderson} §j} 

@heading{ 

name = {Background} fflevel = {level3} author = {Mark Anderson} } 

@heading{ 

name = {The Problem Space}. flevel = {level3} author = {Mark Anderson} } 

@heading{ 

name = {Display in 2D and bitmap (raster) vs. vector formats} fflevel = {level3} {author = {Mark Anderson} §j} 
@heading{ 

name = {The (HTML) Image Map} level = {level3 } author = {Mark Anderson} ,} 

@heading{ 

name = {Raster vs. Vector Data},fjlevel = {level3} author = {Mark Anderson} 4} 

@heading{ 

name = {Issues for Presentation of Infographics in VR} fflevel = {level3} {author = {Mark Anderson} §j} 
@heading{ 

name = {Displaying image data in VR} level = {level3} ffauthor = {Mark Anderson} .j} 


@heading{ 

name = {All surfaces are not web displays},fllevel = {level3} author = {Mark Anderson} 4} 

@heading{ 

name = {What is to be linked and where will the linked resource be found?} [level = {level3} fauthor = {Mark Anderson} §)} 
@heading{ 


name = {Legacy Files—re-mediating pre-existing resources} level = {level3} author = {Mark Anderson}. )} 
@heading{ 

name = {Current files—content designed for combined 2D/3D use} level = {level3} author = {Mark Anderson}. §j} 
@heading{ 

name = {The nature of VR interaction} fllevel = {level3 } author = {Mark Anderson} §} 

@heading{ 

name = {Tool support for linking and re-mediation} level = {level3} author = {Mark Anderson},} 
@heading{ 

name = {Conclusion} fflevel = {level3 } author = {Mark Anderson} ,} 

@heading{ 

name = {Matthias Miller-Prove} {level = {levell } fJauthor = {Matthias Miiller-Prove} ]} 

@heading{ 

name = {On Real and Virtual Text} fflevel = {level2} author = {Matthias Miiller-Prove}.]} 

@heading{ 

name = {From Language to Text} fflevel = {level3} author = {Matthias Miiller-Prove}.]} 


@heading{ 

name = {From Text to Online}, level = {level3} author = {Matthias Miiller-Prove},{} 
@heading{ 

name = {Cool Reading} level = {level3} fjauthor = {Matthias Miiller-Prove},} 
@heading{ 


name = {Hot VR} [level = {level3} fjauthor = {Matthias Miiller-Prove} 4} 
@heading{ 
name = {Real Text in the Virtual World) fflevel = {level3} author = {Matthias Miiller-Prove}.} 


@heading{ 

name = {A Vision for Text in the Virtual World} level = {level3} author = {Matthias Miiller-Prove} } 
@heading{ 

name = {Augmenting Human’s World} level = {level3} author = {Matthias Miiller-Prove}.§} 
@heading{ 


name = {Provisions for the Future} fllevel = {level3 } author = {Matthias Miiller-Prove} 4j} 
@heading{ 

name = {Mez Breeze} level = {level } Jfauthor = {Mez Breeze} 4} 

@heading{ 

name = {Artificial Intelligence Art Generation Using Text Prompts} ,flevel = {level2} {author = {Mez Breeze},{} 
@heading{ 

name = {Beginnings} fflevel = {level3} fjauthor = {Mez Breeze} ,} 

@heading{ 

name = {The Stage} level = {level3} fauthor = {Mez Breeze} 4} 

@heading{ 

name = {The Lowdown} fflevel = {level3} {author = {Mez Breeze} 4} 


@heading{ 

name = {The Impact{s]}} fllevel = {level3} author = {Mez Breeze} 4)} 
@heading{ 

name = {The Rules} level = {level3} author = {Mez Breeze} .j} 
@heading{ 


name = {Conclusions} flevel = {level3},fauthor = {Mez Breeze} 4} 
@heading{ 
name = {Michael Roberts} flevel = {levell } {author = {Michael Roberts}, §} 


@heading{ 

ie = (aetna Combinators: digital tool strategies for the 2020’s and beyond} level = {level2},jauthor = {Michael Roberts} 4} 
@heading{ 

name = {Introduction} fflevel = {level3} author = {Michael Roberts} 4} 

@heading{ 


name = {Programming using node-based languages},level = {level3} author = {Michael Roberts}.)} 
@heading{ 
name = {Combinatorial thinking} filevel = {level3} {author = {Michael Roberts} )} 


@heading{ 

name = {Meta tools} level = {level3} author = {Michael Roberts}. )} 

@heading{ 

name = {Information Hiding} [level = {level3 } author = {Michael Roberts}.4)} 
@heading{ 

name = {Hyperparameters} fflevel = {level3} author = {Michael Roberts}. ]} 
@heading{ 

name = {Machine learning approaches} {level = {level3} {jauthor = {Michael Roberts} } 
@heading{ 

name = {Moving forwards together} level = {level3} {fauthor = {Michael Roberts}, {} 
@heading{ 

name = {Conclusion} level = {level3} fjauthor = {Michael Roberts}. )} 

@heading{ 

name = {Omar Rizwan} level = {levell } author = {Omar Rizwan} 4} 

@heading{ 


name = {Journal : Against ‘text’ } [level = {level2} author = {Omar Rizwan}. j} 


@heading{ 
name = {Patrick Lichty} {level = {levell } ,jauthor = {Patrick Lichty},§} 


@heading{ 

name = {Architectures of the Latent Space} fflevel = {level2} author = {Patrick Lichty}.4]} 
@heading{ 

name = {Context} level = {level3 } author = {Patrick Lichty}.4]} 

@heading{ 


name = {Content} level = {level3} author = {Patrick Lichty},4]} 
@heading{ 
name = {Phil Gooch} level = {level} author = {Patrick Lichty},} 


@heading{ 

name = {Guest Product Presentation : Scholarcy} [level = {level2} fjauthor = {Patrick Lichty} 9} 
@heading{ 

name = {Dialogue} fflevel = {level3} author = {Patrick Lichty},} 

@heading{ 

name = {Peter J. Wasilko} level = {level } author = {Peter J. Wasilko} 4} 

@heading{ 

name = {Benediktine Cyberspace Revisited} level = {level2} author = {Peter J. Wasilko} ,]} 
@heading{ 

name = {Wexelblat’s Taxonomy of Dimensions} fflevel = {level2} {author = {Peter J. Wasilko} 4]} 
@heading{ 

name = {Linnear Dimensions} fflevel = {level3} author = {Peter J. Wasilko}.]} 

@heading{ 


name = {Ray Dimensions} level = {level3} author = {Peter J. Wasilko} } 
@heading{ 
name = {Quantum Dimensions} level = {level3} author = {Peter J. Wasilko} 4} 


@heading{ 

name = {Nominal Dimensions} ‘level = {level3} author = {Peter J. Wasilko},4} 
@heading{ 

name = {Ordinal Dimensions} level = {level3} jauthor = {Peter J. Wasilko} ,]} 
@heading{ 


name = {Functional Dimensions} level = {level3},jauthor = {Peter J. Wasilko},4} 
@heading{ 
name = { Visualizing, Editing, and Navigating Benediktine Cyberspaces} fflevel = {level2} author = {Peter J. Wasilko} 4} 


@heading{ 

name = { Visualization} level = {level3} author = {Peter J. Wasilko},} 
@heading{ 

name = {Editing} level = {level3} fjauthor = {Peter J. Wasilko},9} 
@heading{ 


name = {Navigation} {level = {level3} {author = {Peter J. Wasilko}.]} 
@heading{ 
name = {Comparing Objects} fflevel = {level3} author = {Peter J. Wasilko} 4} 


@heading{ 

name = {The DataProbe HUD — An Additional Possiblity in VR} fflevel = {level2} author = {Peter J. Wasilko} §)} 
@heading{ 

name = {Future Work} fflevel = {level2} author = {Peter J. Wasilko},} 

@heading{ 

name = {Peter J. Wasilko} level = {levell } author = {Peter J. Wasilko} 4} 

@heading{ 

name = {Putting It All Together} level = {level2} fjauthor = {Peter J. Wasilko},]} 

@heading{ 

name = {Interface Affordances for the Serious Use of VR} fflevel = {level3} author = {Peter J. Wasilko}.4)} 
@heading{ 

name = {Future VR Systems Should Embody The Elements of Programming} {level = {level3} fjauthor = {Peter J. Wasilko},4} 
@heading{ 


name = {Requisite Affordances for Productive Work in VR} level = {level3} author = {Peter J. Wasilko}.4)} 
@heading{ 
name = {The VR Pane} fflevel = {level3},{author = {Peter J. Wasilko} {]} 


@heading{ 

name = {The Transcript Pane} {level = {level3} {author = {Peter J. Wasilko}.4]} 

@heading{ 

name = {The Command Line Interface Pane}, [level = {level3 } ffauthor = {Peter J. Wasilko}.} 
@heading{ 


name = {Viewspecs} [level = {level3 } author = {Peter J. Wasilko}.} 

@heading{ 

name = {What Can We Specify with Viewspees?} [level = {level3} fjauthor = {Peter J. Wasilko},{} 
@heading{ 

name = {Examples of Driving Complex Visualizations with a Command Line Viewspee Domain Specific Language (DSL)} level = {level3} author = {Peter J. Wasilko}.4]} 
@heading{ 

name = {UI Support for Discovery of the Viewspec DSL} fflevel = {level3} author = {Peter J. Wasilko} ]} 
@heading{ 

name = {The Gestalt We Are Aiming At} fllevel = {level3} author = {Peter J. Wasilko} 4} 

@heading{ 

name = {Bibliography} level = {level3} fjauthor = {Peter J. Wasilko},]} 


@heading{ 

name = {Pol Baladas},fjlevel = {levell } §Jauthor = {Pol Baladas}.]} 

@heading{ 

name = {There are two great points to be shared after our practical explorations: } fflevel = {level2} author = {Pol Baladas} j} 
@heading{ 


name = {Sam Brooker} [level = {level } author = {Sam Brooker} ,§} 
@heading{ 
name = {Supplementary Material: Devaluing the Work and Elevating the Worker} level = {level2} author = {Sam Brooker}, } 


@heading{ 

name = {Scott Rettberg} evel = {levell } author = {Scott Rettberg} 4} 

@heading{ 

name = {Cyborg Authorship: Humans Writing with AI} {level = {level2} Jjauthor = {Scott Rettberg}.{)} 
@heading{ 


name = {Yiliu Shen-Burke} level = {levell } author = {Yiliu Shen-Burke},} 
@heading{ 
name = {Introducing Softspace} fflevel = {level2} author = {Yiliu Shen-Burke} j} 


@heading{ 

name = {I. Introduction} level = {level3} author = {Yiliu Shen-Burke},} 
@heading{ 

name = {II. Design} fflevel = {level3} {author = {Yiliu Shen-Burke}j} 
@heading{ 


name = {III. User} fflevel = {level3} author = {Yiliu Shen-Burke} )} 
@heading{ 
name = {IV. Flow}, llevel = {level3} author = {Yiliu Shen-Burke} 4} 


@heading{ 

name = {Yiliu Shen-Burke}. level = {levell } author = {Yiliu Shen-Burke},§} 

@heading{ 

name = {Journal Guest Presentation : Discussing Sofispace} {level = {level2},§jauthor = {Yiliu Shen-Burke} ,]} 
@heading{ 

name = {Yohanna Joseph Waliya},fflevel = {levell } author = {Yohanna Joseph Waliya} 4} 

@heading{ 

name = {Post Digital Text (PDT) in Virtual Reality (VR)} level = {level2} author = {Yohanna Joseph Waliya} 4j} 
@heading{ 

name = {Stephen Fry},llevel = {levell } author = {Stephen Fry} 4} 

@heading{ 

name = {In closing: A Prediction} {level = {level2} jauthor = {Stephen Fry} ]} 

@heading{ 


name = {Graffiti Wall} fflevel = {level } {author = {Stephen Fry} .4)} 
@heading{ 
name = {Tom Standage} level = {level2} fauthor = {Tom Standage} 4} 


@heading{ 

name = {Martin Tiefenthaler} fllevel = {level2} author = {Martin Tiefenthaler} 4} 
@heading{ 

name = {Ken Perlin} level = {level2} ffauthor = {Ken Perlin},§} 

@heading{ 

name = {Berard Vatant} fllevel = {level2} {author = {Bernard Vatant},§} 
@heading{ 


name = {Stephanie Strickland} fflevel = {level2} author = {Stephanie Strickland} ]} 


@heading{ 
name = {Anne-Laure Le Cunff} level = {level2} fjauthor = {Anne-Laure Le Cunff}.q)} 


@heading{ 

name = {Stephan Kreutzer} level = {level2} Jauthor = {Stephan Kreutzer} 4} 
@heading{ 

name = {Phil Gooch} level = {level2} author = {Phil Gooch) ,} 

@heading{ 


name = {David Lebow},fllevel = {level2} author = {David Lebow},} 
@heading{ 
name = {Jim Strahorn} level = {level2} author = {Jim Strahorn}.]} 


@heading{ 

name = {Esther Wojcicki} [level = {level2} jauthor = {Esther Wojcicki}.j} 
@heading{ 

name = {Cynthia Haynes} flevel = {level2} author = {Cynthia Haynes} } 
@heading{ 

name = {Peter Wasilko} level = {level2}, author = {Peter Wasilko} §)} 
@heading{ 

name = {Barbara Tversky} level = {level2} author = {Barbara Tversky}.4]} 
@heading{ 

name = {Michael Joyce},fflevel = {level2} author = {Michael Joyce}.{)} 
@heading{ 

name = {Denise Schmandt-Besserat} level = {level2} author = {Denise Schmandt-Besserat}.} 
@heading{ 


name = {David Jay Bolter} {level = {level2} §Jauthor = {David Jay Bolter} 4} 
@heading{ 
name = {Charlie Hargood} level = {level2} author = {Charlie Hargood} 4)} 


@heading{ 

name = {Jonathan Finn} filevel = {level2} author = (Jonathan Finn} 4} 
@heading{ 

name = {Johannah Rodgers} {level = {level2} author = {Johannah Rodgers} )} 
@heading{ 


name = {Dene Grigar},§level = {level2},§author = {Dene Grigar},§} 
@heading{ 
name = {John Cayley} fllevel = {level2} {author = {John Cayley} ]} 


@heading{ 

name = {Alan Laidlaw}. fllevel = {level2} author = {Alan Laidlaw},9} 
@heading{ 

name = {Twitter Comments} level = {level2} author = {Alan Laidlaw}.§)} 
@heading{ 


name = {Conversations from the Journal} level = {level } Jauthor = {Alan Laidlaw) 4} 

@heading{ 

name = {Conversation: Adam’s Experiment} level = {level2} author = {Alan Laidlaw} ,]} 

@heading{ 

name = {Conversation: Experiments with Bob Horn Mural} level = {level2} author = {Alan Laidlaw} 4)} 


@heading{ 

name = {Brandel’s Mural} level = {level2} author = {Alan Laidlaw} ,} 

@heading{ 

name = {Adam Mural with Extracted Dates} level = {level2} {author = {Alan Laidlaw} 4} 
@heading{ 


name = {Conversation: USD (Universal Scene Description)} level = {level2} author = {Alan Laidlaw} 4} 
@heading{ 

name = {History of Text Timeline} fflevel = {level } ,fauthor = {Frode Hegland} 4} 
@heading{ 

name = {13,8 Billion Years Ago} {level = {level2} author = {Frode Hegland) 4} 
@heading{ 

name = {250 Million-3,6 Million} level = {level2} author = {Frode Hegland} 4} 
@heading{ 

name = {2,000,000-50,000 BCE} level = {level2} author = {Frode Hegland}.4} 
@heading{ 

name = {50,000-3,000 BCE} fflevel = {level2},§fauthor = {Frode Hegland} j} 
@heading{ 

name = {4000 BCE} {level = {level2} sjauthor = {Frode Hegland},§} 

@heading{ 

name = {3000 BCE} level = {level2} sjauthor = {Frode Hegland},{} 

@heading{ 

name = {2000 BCE} level = {level2} sjauthor = {Frode Hegland},§} 


@heading{ 

name = {1000 BCE} level = {level2} fjauthor = {Frode Hegland},§} 
@heading{ 

name = {0 CE} level = {level} jauthor = {Frode Hegland)},§} 
@heading{ 


name = {100 CE},level = {level2} author = {Frode Hegland} 4} 
@heading{ 
name = {200} level = {level2} {author = {Frode Hegland} 4} 


@heading{ 
name = {300} level = {level2} ffauthor = {Frode Hegland}.)} 
@heading{ 
name = {400} fflevel = {level2} ffauthor = {Frode Hegland} 4} 
@heading{ 
name = {500} fflevel = {level2} ffauthor = {Frode Hegland} 4} 
@heading{ 
name = {600} level = {level2} {author = {Frode Hegland} 4} 
@heading{ 
name = {700} fflevel = {level2} author = {Frode Hegland}.)} 
@heading{ 
name = {800} level = {level2} ffauthor = {Frode Hegland} 4} 
@heading{ 


name = {900} level = {level2} §jauthor = {Frode Hegland}.} 
@heading{ 
name = {1000} level = {level2} author = {Frode Hegland) 4} 


@heading{ 
name = {1100} fflevel = {level2} {author = {Frode Hegland} 4} 
@heading{ 
name = {1200} level = {level2} fauthor = {Frode Hegland) 4} 
@heading{ 


name = {1300} level = {level2} {author = {Frode Hegland) 4} 
@heading{ 
name = {1400} level = {level2} author = {Frode Hegland) 4} 


@heading{ 
name = {1500} level = {level2} fauthor = {Frode Hegland) 4} 
@heading{ 
name = {1600} level = {level2} fauthor = {Frode Hegland) 4} 
@heading{ 


name = {1700} level = {level2} {author = {Frode Hegland) 4} 
@heading{ 
name = {1800} {level = {level2} {author = {Frode Hegland) 4} 


@heading{ 
name = {1810} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 
name = {1820} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 
name = {1830} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 
name = {1840} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 
name = {1850} {level = {level3} {author = {Frode Hegland) 4} 
@heading{ 
name = {1860} {level = {level3} fauthor = {Frode Hegland) 4} 
@heading{ 
name = {1870} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 


name = {1880} {level = {level3} fauthor = {Frode Hegland) 4} 


@heading{ 
name = {1890} {level = {level3} {author = {Frode Hegland) }} 


@heading{ 

name = {1900} {level = {level2} {author = {Frode Hegland) 4} 
@heading{ 

name = {1910} level = {level3} fauthor = {Frode Hegland) 4} 
@heading{ 

name = {1920},fflevel = {level3} author = {Frode Hegland} } 
@heading{ 

name = {1930} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 

name = {1940} {level = {level3} {author = {Frode Hegland) 4} 
@heading{ 

name = {1950} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 

name = {1960} level = {level3} author = {Frode Hegland} 4} 
@heading{ 

name = {1970} level = {level3} fauthor = {Frode Hegland) }} 
@heading{ 

name = {1980} {level = {level3} author = {Frode Hegland) 4} 
@heading{ 

name = {1990} level = {level3} fauthor = {Frode Hegland) 4} 
@heading{ 

name = {2000} level = {level2} author = {Frode Hegland} } 
@heading{ 

name = {2010} level = {level3} fauthor = {Frode Hegland) 4} 
@heading{ 

name = {2020} level = {level3} {author = {Frode Hegland) 4} 
@heading{ 

name = {Future} level = {level3} {author = {Frode Hegland} 4} 
@heading{ 

name = {Contributors to the Timeline} level = {level3} author = {Frode Hegland} } 
@heading{ 

name = {Colophon} fflevel = {level} author = {Frode Hegland} 4} 
@heading{ 

name = {Gallery from the Symposium} level = {levell } fjauthor = {Frode Hegland} 4} 
@heading{ 

name = {Glossary} level = {level} {showInFind = {false}. §} 
@heading{ 


name = {Endnotes} level = {levell} 4} 

@heading{ 

name = {References},flevel = {level },§} 
@heading{ 

name = { Visual-Meta Appendix}, level = {level },4} 


@ {document-headings-end} 
@{paraText-start} 


@paraText{ 
glossary = {Glossary} §Jendnotes = {Endnotes} references 


{References} fjvisual-meta = { Visual-Meta Appendix} .j} 


@ {paraText-end} 


@{visual-meta-end} 


